I just started auditing a Mathematical Models in Biology class and Matlab is one of the requirements. I had relatively good experience with the free, open source alternative, Octave back in college, but then I was running Linux, not OSX. It took me about an hour to figure out how to set it up (I was a little worried for a bit).

Download the Octave binary for OSX from Octaveforge.

Install Octave and Gnuplot (in the extras folder). I just dragged them to /Applications (X11 is required for Gnuplot—should be found on OSX install disk)

If you are using OSX 10.6 (Snow Leopard) or 10.5.8+ you may need to perform some additional steps outlined here

Set the environment variable for gnuplot (Octave is supposed to do this automatically, but it didn’t for me): `
sudo ln s /Applications/GnuPlot.app/Contents/Resources/bin/gnuplot /usr/bin/gnuplot
`
(thanks for the help, Toby)

Download and install (again in /Applications) Aquaterm which will actually render the gnuplot graphs.

Within Gnuplot, set the renderer: “terminal aqua”

Try it out in Octave (I had to restart Octave and Gnuplot to get it all to work): `
x = linspace(pi, pi, 100);
y = sin(x);
plot(x, y);
`
Thank you: High Performance Computing for Mac OS X, the Octave Wiki and Google for helping me find what I needed.
Update (January 24, 2010): updated the link in step #1 to the latest version of Octave. Added an additional step described by Zack in the comments (thanks!)