You are developing in Python on macOS and want to use pylibmc to access memcached as fast as possible. Sadly, the installation process fails. "'libmemcached/memcached.h' file not found" is the dreaded error message. Below I show the most common causes for this in macOS development environment setups and how to solve them. We'll go from easiest to hardest to solve.

Note that because Linux and macOS can be quite similar, this guide might also help on distributions such as Ubuntu and Debian.

The issue

First, let's look at the error message that your pip install (or poetry or pipenv) spits out:

running build_ext
building '_pylibmc' extension
creating build/temp.macosx-11-x86_64-cpython-39
creating build/temp.macosx-11-x86_64-cpython-39/src
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk -DUSE_ZLIB -I/Users/luis/.local/share/virtualenvs/tack-ng-EDR_dI3-/include -I/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/_pylibmcmodule.c -o build/temp.macosx-11-x86_64-cpython-39/src/_pylibmcmodule.o -fno-strict-aliasing -std=c99
    In file included from src/_pylibmcmodule.c:34:
        src/_pylibmcmodule.h:42:10: fatal error: 'libmemcached/memcached.h' file not found
        #include <libmemcached/memcached.h>
        1 error generated.
        error: command '/usr/bin/clang' failed with exit code 1
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

There's a lot going on here. The rule here is that the most important line in such error messages is usually kept visually distinct and close to the end of the output. In our case we can see some ASCII art arrow:

#include <libmemcached/memcached.h>

Looking a bit closer at the lines around these two, we notice the error message "file not found":

In file included from src/_pylibmcmodule.c:34:
    src/_pylibmcmodule.h:42:10: fatal error: 'libmemcached/memcached.h' file not found
    #include <libmemcached/memcached.h>
    1 error generated.
    error: command '/usr/bin/clang' failed with exit code 1

Ok, so whatever pylibmc is trying to do, it can't because it can't find libmemcached.

libmemcached really is not installed

You actually do not have libmemcached installed. Oops! Let's fix this:

# using your package manager of choice. On macOS with homebrew:
$ brew install libmemcached
# or you are running some Linux:
$ apt install libmemcached-dev

# After that is done, retry installing via pip (or poetry or pipenv; whatever floats your boat)
$ pip install pylibmc

You are still seeing the dreaded "libmemcached not found" message? Read on. We're not done yet. There's one more thing we can try.

Libmemcached is in a non-standard location

This is by far the most common cause for any such issue were header files can not be found. There is no magic way of asking a system for the location of all libraries. If your system (or package manager) does not, or can not, adhere to standard paths, the chance of failure during compilation are high.

Now, how do we find the location of our libmemcached and tell that to pylibmc during installation?

We ask our package manager!

$ brew info libmemcached
libmemcached: stable 1.0.18 (bottled)
C and C++ client library to the memcached server
/usr/local/Cellar/libmemcached/1.0.18_2 (231 files, 2.2MB) *
  Poured from bottle on 2021-01-04 at 13:57:45
License: BSD-3-Clause

Here we go: /usr/local/Cellar/libmemcached/1.0.18_2

Note though that your path might look different. If you are running Homebrew on a Mac with Apple Silicon, this path will be very different. Are you using Macports? yep, different. On Linux? Also different. How can you find the path to libmemcached then? Here's some shell magic for you:

# Look for the missing file mentioned in the pylibmc install output.
# Then we'll use the most general looking path (the last one)
$ find /usr -name 'memcached.h' -type f

Let's use our new found knowledge to hopefully finally install pylibmc:

# Use the path we previously looked up.
# if your shell is bash/zsh:
$ LIBMEMCACHED=/usr/local/Cellar/libmemcached/1.0.18_2 pip install pylibmc
# or on fish:
$ env LIBMEMCACHED=/usr/local/Cellar/libmemcached/1.0.18_2 pip install pylibmc

This should have finally worked. If it did not, there is still one little possibility left.

Libmemcached does not match a supported version

You might be running an old OS. In such a case, it is hard to give specific guidance. However, here is some general guidance:

  1. Find out if there is a way to install a newer version of pylibmc and libmemcached
  2. If there is not and your pylibmc is fairly old, try to find a way to install an older version of libmemcached
  3. If your pylibmc is a newer release but you can't get a modern libmemcached, you will probably have to compile libmemcached yourself.
  4. As a last resort: are you sure you need pylibmc? There are also python only memcached clients. Maybe one of those would be a good fit as well.

I hope this post helped you out. Did I miss another common case? Let us know on Twitter.

Other Posts

  • When Homebrew's Postgres Upgrade Fails

    Internally we upgraded our development machines from Postgresql 12 to version 13. Things went sideways because we also use Postgis in a lot of our projects. This is what to do when this happens.

  • Create wagtail pages programmatically

    Programmatically creating the Wagtail page tree can save you and your team a lot of time in the long run.