Build System Caveats¶

All LSST Data Management packages that follow the standard template and are integrated into the EUPS build system, use SCons for building. SCons does not yet work natively on Python 3. This means that SConstruct, SConscript and sconsUtils .cfg files should not assume that the Python running the SCons code is the same Python as that being used to run the stack code. This means that decisions should not be made by querying distutils.sysconfig or sys.executable within the code. Instead, the python on $PATH must be queried instead, using subprocess.

Code cleanup¶

Many of the Python files will be modified as part of the port and there will be extensive testing during the port itself. It is therefore an excellent time to run a code cleanup enabled by the redefinition of the DM Python coding standard in terms of PEP-8. The autopep8 tool can fix many of the whitespace issues present in the codebase. Use the command as follows:

autopep8 {{package_dir}} --in-place --recursive --ignore E26,E133,E226,E228,N802,N803 --max-line-length 110

where {{package_dir}} is the directory that is the directory that is being cleaned up and the ignore list specifies which PEP-8 fixes should not be implemented. This tool will not fix all the style problems with the codebase but it will help a lot with the more basic issues that are common to every file. Use the flake8 tool to get a more extensive report on coding standard violations and consider embedding flake8 support in your favorite editor to get real time feedback on issues.

Modernize¶

The LSST DM code has been under development since 2004 and has a lot of code that was developed when Python was at version 2.3. This means that there are many places in the code that do not use the __future__ construct to enable true division, print function and absolute import (some code is old enough that is has to enable with_statement in Python 2.5) for simple Python 3 compatibility. Also, many exceptions are being caught with catch Exception, e rather than the more modern catch Exception as e. All these constructs are not compatible with the current DM coding standard and should be modernized in a distinct commit.

The easiest way to achieve this is to run the futurize command found in the Python future package. futurize has a two stage conversion option where it is possible to first run modernization to Python 2.7 before attempting to support 2.7 and 3. Run the command as follows:

futurize -1 -n -w .

where -1 runs the stage 1 fixer, -n disables backup files (you are running this in a directory managed by git) and -w indicates that the files should be updated (default is to tell you what would happen). Most of the changes are straightforward and should be applied. The one issue resolves around code that uses the types.StringType constant. This no longer exists on Python 3 and futurize decides that since it’s a synonym for bytes that the entire thing should be replaced by bytes. It is probably more likely that the code should be replaced by str.

Remove deprecated calls¶

In Python 3 some unittest methods have been officially deprecated and should be replaced with modern equivalents. The assert() method should be replaced with assertTrue() and assertEquals() should be replaced with assertEqual(). Many of these changes were implemented as part of the pytest migration (see also SQR-012) so it is likely that these specific issues will not be encountered.

Futurize¶

The final step in supporting Python 2 and Python 3 is to run the futurize command to enable the stage 2 conversion. The following command is recommended:

futurize -2 -x division_safe -n -w .

The -x option here specifically disables the protection of division operators (see below). futurize will modify imports to enable backwards compatibility shims. Imports from builtins will be no-ops on Python 3 but on Python 2 will import versions that behave like Python 3 (in Python 3 __builtins__ was renamed builtins). It is common to see imports for str, zip, object and range added to the top of a file. The conversion is not going to be perfect and should not be accepted without examining the changes using an interactive GUI tool or git add -p. In particular futurize will make the following changes that should be looked at carefully before applying:

• futurize is very defensive in its futurizing. If a construct is being used in Python 2 that returns a list (such as the keys() method on a dict), then futurize will replace that with list(mydict.keys()). Rather than accepting everything, developers should decide whether to accept the list variant on a case-by-case basis. In a loop it is probable that the list should be removed (unless the object is modified in the loop). If the list is being returned from a function it should probably be retained.
• In Python 2 the / division operator can result in integer division depending on the operands, unless __future__ division is enabled. In Python 3 division is always “true” division and the // operator must be used to perform integer division. This operator is also available in Python 2 but by default futurize plays it safe and will replace all divisions with a function call, old_div, that tries to retain the Python 2 default behavior in Python 3. It’s always best bit to present.. Unfortunately, disabling this option prevents futurize from inserting the __future__ division import at the top of the file (this import is required by our coding standard).
• Methods called next() that are not implementing Python 2 iterators will confuse the tool. It assumes that it is an old-style iterator and will rewrite it using the modern next() function. If next() methods are present that are not associated with iterators, consider renaming them to avoid confusion in the future. In Python 3 (and 2.7) the iterator version should be called __next__().
• Python 3 int is the same as Python 2 long and futurize will convert long() to int() during the conversion. It may be necessary to retain a Python 2 long type in the code and future allows this via the past.builtins package. In many cases in the Data Management code it may be that the distinction between int and long is no longer relevant as int in a 64-bit build of Python is a 64-bit integer and many of the cases in the DM code base use long to mean “64-bit” rather than arbitrary precision. This can be seen in the use of the long type in lsst.daf.base.
• future provides a str class that behaves like a Python 3 str but is neither a Python 2 str nor a Python 2 unicode (it is technially a newstr. This works fine when a string is being used as a string but unless unicode_literals is enabled string literals will behave like native str. This will result in type(str) != type("") and isinstance("", str) returning False on Python 2. The correct Python 2 solution for handling unicode and str is to use isinstance(var, basestring) and this is usually the easiest way to write code in Python 2 that can recognize all the string types. Python 3 does not understand this class but the future framework can enable this compatibility using from past.builtins import basestring. This is the simplest solution to compatibility with the three distinct string classes in Python 2 (when future str is included) and is one example where Python 2-style code must be used.

Finishing the port¶

futurize will not completely fix all of the issues with Python 3/2 compatibility. Many of the remaining problems will be related to Python 3 being strict when handling strings and bytes. Python 2 treated bytes and strings as synonyms but one of the key reasons for Python 3 to exist is that it will not use bytes when a string is required.

• Long integer literal (for example 5L) syntax is not supported in Python 3 because all integers are long. If a long int is required in the Python 2 implementation it must be created using the long constructor imported from past.builtins.
• sorted() no longer has a cmp argument and control of the sort order must be changed to use a key function.
• str.translate() changes how it works depending on whether the string is Unicode or bytes. Additionally the string.maketrans() function has been removed from Python 3. Use a regular expression substitution instead.
• Some DM code tries to distinguish a string from a sequence by checking if the __iter__ attribute exists. This does not work on Python 3 because the str class does not support __iter__. An explicit check for a string instance is required.
• In rare cases a class with multiple inheritance will fail on Python 3 because the two parent classes have metaclasses defined that are incompatible. The solution for this is to define a new metaclass that combines both the metaclasses and this may require distinct code paths for Python 2 and 3. An example of the technique to use can be found in lsst.daf.persistence.policy.Policy.

import subprocess
result = subprocess.check_output("ls").decode()

%extend lsst::afw::math::Random {

    PyObject * getState() const {
        std::string state = self->getState();
        return PyBytes_FromStringAndSize(state.data(), state.size());
    }

    PyObject * setState(PyObject * state) {
        char * buffer = nullptr;
        Py_ssize_t len = 0;
        if (PyBytes_AsStringAndSize(state, &buffer, &len) == 0) {
            std::string state(buffer, len);
            self->setState(state);
            Py_RETURN_NONE;
        }
        return nullptr;
    }

}

%ignore lsst::afw::math::Random::getState;
%ignore lsst::afw::math::Random::setState;