Month: February 2021

The problem with knowing people who are very smart is that occasionally you get dragged into their smart-person shenanigans. Dumb people are more fun in this way; their idea of a good time is to go set off all the illegal fireworks they found in a trash bag next to the freeway inside the nearest dumpster or abandoned refrigerator, which is generally a good time and requires little from the average bystander except a willingness to bring their own fire extinguisher and an ability to watch out for the cops. Smart people, however, will drag you into Intellectual Pursuits, and it’s thus that I’ve ended up banging my head obsessively against trying to get Exotic installed on Big Sur.

“What is Exotic”, I hear you cry? Well, it’s a clumsy acronym for EXOplanet Transit Interpretation Code, which in its simplest terms (which are the only terms that I can reliably understand), means that it’s a Python package that you use to reduce photometric data of transiting exoplanets into light curves, thus retrieving transit epochs and planetary radii.

Okay, you got me. I copied that last bit from the Exotic GitHub page. I am not a data scientist; I am merely an well-coiffed ape with a dream and what Liam Neeson would doubtless describe as a set of Very Particular Skills – none of which involve understanding what the hell a “transit epoch” might consist of.

Still, Better Minds Than Mine assure me that Exotic is a Python package that examines data from massive telescopes and interprets that data to let you know if there might be a planet zipping around a distant star, and who am I to challenge this claim? How does it do this arcane task? I don’t know. Hell, I don’t think anyone really knows – this kind of thing, as far as I’m concerned, is essentially synonymous with witchcraft. Which has some tonal resonance for me, because getting the thing installed on a Mac running Big Sur required a certain amount of supernatural wrangling all of its own.

For one thing, it requires a long list of Python dependencies (software components required by a system to be in place before you install The Thing You Really Wanted To Install) to get the thing installed, and most of those dependencies have their own lists of dependencies, and after a while it starts to feel like there’s dependencies all the way down. Some of those dependencies install without error or comment, and others fail in the most dramatic way possible – mostly by spitting out 2,700 lines of alarming red text where the precise shade of red has been carefully engineered to provide maximum guilt and terror (red on black is also hard to read, but I’m going to stick with the guilt/terror thing as that means I don’t have to confront my fading eyesight. Having bad ears is quite enough, thank you).

Apple helpfully updates Python with every OS release, which has the unfortunate effect that they’re continuously moving goalposts without telling anyone – so a package that might work just fine with one release might fail utterly after a seemingly trivial point upgrade. Fortunately, you can get hold of a lot of old versions of Python here. Which is what I did – downloading a fresh copy of Python 3.8.6. This, unfortunately, is the non-M1 version of Python and requires Rosetta 2 to be installed on your M1 Mac, but there’s regrettably not much that can be done about that.†

You can, it should be noted, install as many versions of Python as there are (ha ha) stars in the sky. The problem with that approach is that you’ll possibly need to switch between them using something like pyenv, but in this case it wasn’t important to do a lot of mucking around with keeping options open vis-a-vis Python versions; the prime directive was to make sure that as far as the Mac was concerned when someone typed python then this was interpreted as the freshly-installed Python 3.8.6, and not the included-with-the-computer-and-hard-to-remove Python 3.9.2.

This was fairly simple to accomplish with a command to write an alias into the .zshrc file:

echo "alias python=/usr/local/bin/python3.8" >> ~/.zshrc

What this does is inject the alias python=/usr/local/bin/python3.8 command into the file that macOS uses to configure the zsh shell so that whenever you type “python” into the terminal your Mac sort of clears it’s throat and discreetly directs whatever comes next toward the version of Python that you tell it to instead of the one that it would customarily choose right out of the box.

Next, there are three python packages that should be installed before attempting to install Exotic:

pip3 install wheel

pip3 install importlib_metadata

pip3 install —upgrade keyrings.alt

With those in place, you’ll need to run the Install Certificates command that you should find in the Python 3.8 folder in your Applications folder (otherwise you’ll run into all kinds of SSL/TLS errors. Which are Bad Things).

Finally, you can install exotic:

pip3 install exotic

Note: You may have to run this command twice as I’ve seen it error out on installing one package with a missing dependency that is fixed by a second install.

All things being equal, you’ll be able to type exotic in the Terminal, and see something like this:

† – Way back at the beginning of this article I mentioned that we installed a non-Apple Silicon version of Python (3.8.6), which means that if you’re using an M1 Mac then that M1 Mac will be running Python through emulation – in effect, pretending to be an Intel-based computer. This, it should be noticed, means that it’s running a lot slower than it would be capable of doing with software written for the ARM-based chip in the M1 (Later versions of Python are available that are optimized for Apple Silicon, but none of them seem to work with Exotic). This sounds bad – and in reality it’s not perfect, but here’s some anecdotal feel-good data for you.

The ingenious boffins who came up with exotic also make some test data available so that you can try the thing out to make sure it works. Running that test project on a 2017 i7 MacBook Pro took twenty-two minutes to process and render the test data, accompanied by a chorus of whirring fans and a truly alarming spike in how hot the computer got.

My 12-core Xeon Mac Pro with a bucket of RAM and fast drives took a decently respectable fifteen minutes to chew through the same data and spit out a fancy-looking graph, and because it’s largely comprised of fans (and pretty noisy as a baseline) it didn’t get noticeably hot or bothered by this procedure

A bottom-of-the-line, base spec M1 MacBook Air? Five minutes. It took five minutes, and the thing didn’t so much as get mildly warm. Running on an older version of Python while under emulation. Five minutes.

This is… well. Blimey.

macOS (at least, as of Big Sur 11.2.1) ships with Python 2.7.16 by default. This seems curious when you consider that Python 2.7 was deprecated and no longer supported after December 2020, but there are Reasons for shipping the newest and shiniest macOS operating system with a (much) older version of Python.

For one thing, Python 3 isn’t backward compatible, and Python 2 has vast library support that might be critical to your workflow, and not having those libraries available seems like a recipe for disaster. For another thing, you invoke Python 2 with the python command, and Python 3 with the python3 command – and while this seems less obviously an issue it’s actually a much bigger problem that you’d expect. After all, how many pieces of code or scripts are out there, undocumented, starting with #!/usr/bin/python? I’m betting the answer to that question is, well, a lot.

So, what to do if you want to be able to run both Python 2.7 and Python 3 on a single install of macOS? Well, there’s a two-pronged way of doing it by utilizing the homebrew pyenv package (which is ingenious and well-supported) along with a little .zshrc hack (which is something I came up with and therefore almost certainly deeply problematic).

You can find pyenv here, and the simple installer for it linked here. In the simplest terms, what pyenv does is intercept any request that’s made for python commands and then slides those requests over to the selected version of python. It’s extremely clever and additionally that rare and unusual creature in that it’s a GitHub project that’s both well-maintained and well-documented. You’ll need to install the Xcode Command-Line tools with an xcode-select --install command, but once that’s done you can run the installer linked above and it shouldn’t cause you much trouble.

Once installed, there are a couple of additional required steps. Firstly, you’ll need to make a backup of the .zshrc file in your home directory with ditto ~/.zshrc ~/.zshrcbackup (if you don’t already have a .zshrc file then feel free to skip this part.)

Next, make another copy of your .zshrc file: ditto ~/.zshrc ~/.zshrc2 (This may seem needlessly duplicative, but it’ll come in handy in the ugly hack part of the procedure later on).

Finally, edit your .zshrc file and add these lines onto the end:

export PATH="/Users/daveb/.pyenv/bin:$PATH" (Note: substitute your short username for daveb unless your name is also Dave B.)

eval "$(pyenv init -)"

eval "$(pyenv virtualenv-init -)"

Once you’ve done all the above, quit Terminal, open it again, then install your Python version of choice by issuing the pyenv command, invoking the install option, and then adding the version of Python you want to install like so:

pyenv install 3.9.1

…and that’s it. If you want to install other versions of Python then you can run the pyenv install command again with the relevant build, and if there’s one particular build that you want to, say, specify Python 3.9.1 as the default then you can make that so with the pyenv global 3.9.1 command. A more complete and in-depth list of commands can be found here.

So, now that’s all the pretty, elegant stuff out of the way, so time to move on to the ugly hack. The problem I’ve found with pyenv is that it’s great for installing versions of Python that are up-to-date, but that it really doesn’t seem to want to toggle back to the version that Big Sur ships with, and there doesn’t seem to be a convenient off switch for the thing. Further, trying to install that old version (2.7.16) doesn’t work because Python 2.7.16 is ancient and wicked and pyenv refuses to allow something so arcane to be installed on your machine (despite the fact that it was on there already).

To get around that, I added this to my .zprofile:

alias py='mv ~/.zshrc ~/.zshrctemp; mv ~/.zshrc2 ~/.zshrc; mv ~/.zshrctemp ~/.zshrc2'

Ugh. Just look at that thing.

Like I said, it’s ugly, but when you type the alias py into a terminal window then what happens next is that it takes your existing .zshrc file, renames it to .zshrctemp, then takes the .zshrc2 file and renames it .zshrc, and then finally takes the .zshrctemp file and rename it .zshrc2.

In effect, it takes the .zshrc2 file (the one that isn’t set up to enable pyenv) and switches it out with your .zshrc file (the one that is set up to enable pyenv). When .zshrc2 becomes .zshrc, pyenv suddenly no longer works, and your computer defaults to the version of python that it shipped with (in my case 2.7.16) because it suddenly doesn’t know any better. And because the command just switches those two files around, issuing it again turns the tables and re-enables pyenv.