Detect clones configuration

It is now possible to configure the detect clones functionality for finding duplicate code.

This configuration must be added at the project level in the .sourcery.yaml file, under the clone_detection heading. See here for more details on the Sourcery configuration file.

• min_lines: The minimum number of lines each code section must have to be picked up. The minimum value for this is 3.
• min_duplicates: The minimum number of duplicate code sections there must be. The minimum value for this is 2.
• identical_clones_only: When this is set only exactly identical sections of duplicate code will be picked up.

clone_detection:
  min_lines: 3
  min_duplicates: 2
  identical_clones_only: false

Working memory metric improvements

The working memory metric will now only consider the most complex element within collections or call arguments, rather than incrementing for each element.

This means that when declaring a list like this one:

hats = ['BOWLER', 'TOP' + 'HAT', 'TRILBY']

The working memory for this statement will now be 3 (incrementing for hats, TOP and HAT) whereas before it would have been 5. This will prevent the phenomenon where creating dictionaries, lists or data classes would give functions unreasonably high working memory scores. For a full discussion of the working memory metric see our blog post here.

New refactorings

Swap Variable

Sourcery refactoring id: swap-variable

Description:

Swap variable values with tuple assignment

Before:

temp = a
a = b
b = temp

After:

a, b = b, a

Explanation:

When trying to swap the values held by two variables, we may avoid using an extra temporary variable unnecessarily by employing Python's tuple assignment. As a consequence of this refactoring, the code is not only more concise but also reflects its intention more clearly.

Merge Else If Into Elif

Sourcery refactoring id: merge-else-if-into-elif

Description:

Merge else clause's nested if statement into elif

Before:

def interpret_response(response):
    if response.status == '200':
        return response.data
    else:
        if response.status == '404':
            return 'Not Found'
        else:
            return 'Error'

After:

def interpret_response(response):
    if response.status == '200':
        return response.data
    elif response.status == '404':
        return 'Not Found'
    else:
        return 'Error'

Explanation:

Flattening if statements nested within else clauses generates code that is easier to read and expand upon.

Inline variable

Sourcery refactoring id: inline-variable

Description:

Inline variable that is only used once

Before:

thelist = []
for i in range(10):
    k = i ** 2
    thelist.append(k)

After:

thelist = []
for i in range(10):
    thelist.append(i ** 2)

Explanation:

Inlining variable can help to streamline the code, but can also make it less readable. Sourcery will only inline variables where doing so allows further readability changes to be made, such as converting a loop into a list comprehension.

Remove redundant continue

Sourcery refactoring id: remove-redundant-continue

Description:

Remove redundant continue statement

Before:

mylist2 = []
for i in mylist:
    if i != 2:
        mylist2.append(i)
    else:
        continue

After:

mylist2 = []
for i in mylist:
    if i != 2:
        mylist2.append(i)

Explanation:

If a continue is not followed by any other statements in a for or while loop then it is not necessary, so can be removed. Removing unnecessary lines declutters the code and makes it easier to understand. This refactoring will only be triggered if it unlocks further improvements.

Reintroduce else

Sourcery refactoring id: reintroduce-else

Description:

Lift code into else after break in control flow

Before:

summer_hats = []
for hat in hats:
    if hat in WINTER_HATS:
        continue
    summer_hats.append(hat)

After:

summer_hats = []
for hat in hats:
    if hat in WINTER_HATS:
        continue
    else:
        summer_hats.append(hat)

Explanation:

Where the body of an if statement ends with a break in the control flow, such as a continue, return or raise, the subsequent statements can be lifted into the else clause. On its own this change does not improve the code, so Sourcery will only suggest it where it unlocks furher improvements. In the example above once the code has been lifted into the else the conditional can be inverted and the continue removed, which then lets the for loop be converted into a list comprehension.

Remove Duplicate Key

Sourcery refactoring id: remove-duplicate-key

Description:

Remove duplicate keys in declaration of sets and dicts

Before:

addresses = {*address_list1, *address_list2, *address_list1}

After:

addresses = {*address_list2, *address_list1}

Explanation:

Keys of Sets and Dictionaries must be unique. Hence, repeated keys are redundant and can be removed from the declaration to increase the conciseness and clarity of the code.

For dictionaries that have repeated keys with differing values (i.e. {key1: 'some_value', key1: 'another_value'}), Sourcery will only remove the key-value pairs that would be removed at run time.

Flatten Nested Try

Sourcery refactoring id: flatten-nested-try

Description:

Merge nested try-statement into a single try

Before:

def testConnection(db, credentials):
    try:
        try:
            db.connect(credentials)
        except InvalidCredentials:
            return 'Check your credentials'
        except ConnectionError:
            return 'Error while trying to connect'
    finally:
        print('Connection attempt finished')
    return 'Connection Successful'

After:

def testConnection(db, credentials):
    try:
        db.connect(credentials)
    except InvalidCredentials:
        return 'Check your credentials'
    except ConnectionError:
        return 'Error while trying to connect'
    finally:
        print('Connection attempt finished')
    return 'Connection Successful'

Explanation:

Flattening try-except statements nested within a try-finally generates equivalent code that is easier to read and expand upon.

Collection into Set

Sourcery refactoring id: collection-into-set

Description:

Use set when checking membership of a collection of literals

Before:

if currency in ['EUR', 'USD']:
    take_payment()

After:

if currency in {'EUR', 'USD'}:
    take_payment()

Explanation:

When checking if a variable is one of a collection of literals it is both more natural and more performant to use a set rather than a list or tuple to define that collection. Note that this can only be done where Sourcery can ascertain that the value being checked is of a hashable type.

Enhancements

• Sourcery will now suggest refactorings which involve the removal of comments. This allows us to make more suggestions and be more impactful on your code. These suggestions are flagged and the comments that are removed are clearly shown when hovering over.
• Simplify generators contained in calls to bytes
• Adjust return-identity proposal so that it does not propose in situations where a series of checks are being made.
• Flip Compare now works for all constant types
• Merge comparison refactoring extended to work with sets and tuples
• Ensure switch proposal is not suggested where the if conditions end with breaks of control flow
• Refinements to where the else-after-guard proposal is suggested

Bug fixes

• Fixed issue with Sourcery incorrectly removing brackets after a splat operator

New Refactorings

Replace dict.get(x, None) with dict.get(x)

Before:

hats = {"bowler": Bowler(), "sombrero": Sombrero()}
fedora = hats.get("fedora", None)

After:

hats = {"bowler": Bowler(), "sombrero": Sombrero()}
fedora = hats.get("fedora")

Explanation:

When using a dictionary's get method you can specify a default to return if the key is not found. This defaults to None, so it is unnecessary to specify None if this is the required behaviour. Removing the unnecessary argument makes the code slightly shorter and clearer.

Remove unit step from range

Sourcery refactoring id: remove-unit-step-from-range

Description:

Replace range(x, y, 1) with range(x, y)

Before:

for i in range(y, len(x), 1):
    do_t()

After:

for i in range(y, len(x)):
    do_t()

Explanation:

The default step value for a call to range() is 1, so it is unnecessary to explicitly define it. This refactoring removes this argument, slightly shortening the code.

Small fixes and enhancements

• Enhance removal of redundant exception handlers to support tuples
• Improved installation instructions for VS Code
• Fix for issue where conditionals could be removed incorrectly
• Fix issues with internal tracking metrics

Duplicate code detection in PyCharm

Our new duplicate code detection feature is now available for PyCharm!

You can now select multiple files or folders and choose 'Sourcery - Detect Clones' to scan for duplicated code within them. This will find sections of code that are 3 lines long or greater that are duplicated in multiple places. It will also find near-duplicate code where variable names have been changed but the code structure remains the same. The duplicate code found will appear in the Sourcery tool window.

This feature is available in our Pro subscription. It is in active development so we would love your feedback.

Disabling code quality warnings

Have a piece of code that you don't want to touch that is giving a low quality warning?

You can now disable these warnings in VS Code and PyCharm. Add # sourcery no-metrics at the end of the line with the function definition to do so. There's also a new quick-fix action on the warnings to disable them.

Minor fixes

• Fixed issue where Sourcery scan commands were incorrectly showing in VS Code command palette
• Running a scan for refactorings in VS Code now shows a link to the problems pane in the notification that the scan is complete.

Enhancements

• Added links to documentation from refactoring hover
• Added local logging to plugins to improve diagnosis of issues going forward

Bug fixes

• Fixed issue where plugins were eating too much memory
• Improved performance of CLI
• Fixed issue with incorrect comprehension proposal for async for loops.

Sourcery Pro launched

You can now purchase a Sourcery Pro subscription.

With Sourcery Pro you can:

• Remove duplicate code with our new automated method extraction
• Refactor your whole project in the IDE
• Use our GitHub bot on private repos
• Find duplicate code across your project (VS Code only at present)

Get more details here.

Duplicate code detection

You can now select multiple files or folders and scan for duplicated code within them. This will find sections of code that are 3 lines long or greater that are duplicated in multiple places. It will also find near-duplicate code where variable names have been changed but the code structure remains the same.

This feature is available in our Pro subscription, for VS Code users only initially. It is in active development so we would love your feedback.

Sublime support

Sourcery is now available in Sublime.

Get more details and setup instructions here.

New refactorings

Use str.join() instead of for loop: use-join

This will convert code like this:

result = ""
for text in long_list_of_text():
    result = result + text
return result

into:

result = "".join(long_list_of_text())
return result

Simplify Constant Sum: simplify-constant-sum

Changes this:

results = sum(1 for token in source.split() if token == 1)

to this:

results = sum(token == 1 for token in source.split())

Minor fixes

• Make simplify-boolean-comparison suggest more often
• use-count now triggers on the result of simplify-constant-sum rather than directly
• remove-redundant-if now slightly more powerful
• Improved handling of if..elif chains with repeated code
• Don't remove-redundant-if when invalidated by a context manager
• Fix issue where switch refactoring could produce incorrect result
• Fix issue with hoisting statements from loops which could produce incorrect results
• Fixed issue in PyCharm where Sourcery could lose connection to binary when it was scanning a very large function.

Improved low code quality warning message

This message now displays the name of the function where we have detected low code quality. This lets you easily see at a glance where the problems in your code are.

Bug fixes

• Improved PyCharm stability when viewing files with large numbers of functions
• Fix indentation issue when refactoring functions inside classes in PyCharm
• Fix error in GitHub bot quality report where section of code has zero weighting

Extract Method Refactoring

Sourcery can now propose refactorings to extract blocks of code into new functions. This currently proposes in two circumstances:

• When finding duplicate and near duplicate blocks of code within a single function
• When finding a large block of code in the function that would be better off in it's own function

This feature will be available in the upcoming Pro subscription.

Full project scan for refactorings

Select the project or a folder within a project in the IDE plugins, and then have Sourcery scan every file within it. All refactoring suggestions will be displayed as they are found in the Problems window.

This makes it easy to refactor a large portion of the codebase quickly without manually having to visit every file to see the Sourcery suggestions.

This feature will be available in the upcoming Pro subscription.

Improve project and module code metrics in Quality Report

Previously when aggregating metrics we simply calculated metrics for each function and then averaged them. This did not account for the differing size of functions - a 100 line function has more impact on quality than a 10 line one.

We now weight each metric (apart from method length) by the number of lines in each method when averaging them. This gives a better view of the actual file and project-level quality.

New Refactorings

Hoist code from else after guard clause: remove-else-after-guard

Hoist code from an else after a guard clause. This will happen where the main body of an if contains only a raise or return statement, and there are multiple statements in the else.

def f(a=None):
    if a is None:
        return 42
    else:
        # some long statement
        var = (i ** 2 for i in range(a))
        return sum(var)

is converted into:

def f(a=None):
    if a is None:
        return 42

    # some long statement
    var = (i ** 2 for i in range(a))
    return sum(var)

Inline variable that is immediately returned after if

Where a value is set on each branch of an if and then immediately returned, instead return it directly from each branch.

def f():
    if condition:
        val = 42
    else:
        val = 0
    return val

is converted into:

def f():
    if condition:
        return 42
    else:
        return 0

Add Command Line Interface --backup option

Use with sourcery refactor --in-place to backup changed files with given suffix:

sourcery refactor --in-place --backup .bak main.py

If any refactorings are found in main.py a copy will be made at main.py.bak before it is updated in place.

Bug Fixes and other adjustments

• Only propose for-index-replacement for lists, as it's possible __len__ and __getitem__ are implemented but __iter__ is not.
• Minor changes to the swap-if-else refactoring. This will now also be triggered where it can enable the above remove-else-after-guard refactoring. The description has also changed to reflect this.
• Fix issue where block comments above a changed line stopped refactorings being suggested
• We now don't convert an if/else to an if expression where the test contains a boolean operation. This prevents creation of hard-to-read if expression lines.
• Quality warnings in PyCharm Problems pane no longer show as raw HTML
• IDE Code metrics now have link to our documentation. We have added a title to the code metrics in the IDE with a link to our documentation. This will let you easily see how the metrics are calculated and how to improve your code.
• Fix formatting issue in Vim where a blank line was deleted after the refactored function

New Refactorings

Simplify negative list access

This refactoring uses the fact that Python allows negative list indices to be accessed directly.

It converts this:

last_element = a[len(a) - 1]

into this:

last_element = a[-1]

Merge Comparison refactoring extended to the negative case

The merge-comparison proposal will now apply in this case:

if x != a and x != b:
    do_stuff()

This will be converted to:

if x not in [a, b]:
    do_stuff()

Bug fixes

• Ensure statements that write global state can't be hoisted out of loops (e.g. function calls)
• Do not try to remove pass statements at class level
• Do not wrap the targets of annotated assignments in brackets

Sourcery CLI

Sourcery is now available as a command line interface. This enables several new use cases:

• Refactor files on your machine without needing to run Sourcery through PyCharm or VSCode
• Check every commit is refactored using a pre-commit hook
• Run a continuous integration job to check all code is refactored

This functionality is only available with Pro/Team subscriptions. If you'd like to try it out for your team please contact us.

Installation

The Sourcery command line interface can be installed by running:

pip install sourcery-cli

Login

Once installed you can interactively login with:

sourcery login

which will open up a browser tab and ask for confirmation.

Usage

To display suggested refactorings as a diff:

sourcery refactor {file_or_directory}

And to apply those changes to the files:

sourcery refactor --in-place {file_or_directory}

Full documentation is available here.

Code quality metrics enabled by default in the IDE plugins

You can now hover over a method definition to see a quick view of its code quality. The metrics available are:

• Cognitive complexity - a measure of how much complex nested logic is in the method
• Size - a measure of how large the method is
• Working memory - a measure of how many variables you need to keep in mind to understand the most complex parts of the method
• Quality - a combination of the above metrics to give a percentage quality score

This can be switched off in the Sourcery section of the plugin settings.

New Refactorings

Use with context manager to ensure file closure

A common way of opening and using files is:

file = open("welcome.txt")
data = file.read()
print(data)
file.close()

However if an exception is thrown in between the file being opened and closed the call to file.close() may end up being skipped. By using Python's with context manager the file is closed for you as soon as the block is exited.

with open("welcome.txt") as file:
    data = file.read()
    print(data)

Extending the list-comprehension refactoring

We have now extended this refactoring to include an additional case, where augmented assignment is used to add to the list rather than append.

The following code:

files = []
for x in file_iterator:
    if x[-4:] == ".csv":
        files += [x]

will now be refactored as:

files = [x for x in file_iterator if x[-4:] == ".csv"]

Simplify if expression by using or

Often we find ourselves setting a value if it evaluates to True, and otherwise using a default.

currency = args['currency'] if args['currency'] else DEFAULT_CURRENCY

This can be simplified to the following, which is a bit easier to read and avoids the duplication of args['currency'].

currency = args['currency'] or DEFAULT_CURRENCY

Bug Fixes

• Prevent use-assigned-variable from re-using properties

Enhanced code quality metrics in the IDE

We've been making some changes to how our code quality metrics are displayed.

When hovering over a method definition you now get more explanation of the code metrics. This means that you can see if the metric scores are good or bad at a glance.

In VS Code this is an emoji as shown above, whereas in PyCharm it is shown in writing. For each metric if the score is below average we also give a tip on how to improve it.

Right now these metrics are hidden by default in PyCharm and VS Code, but they're very easy to enable.

To set them up just add the following setting to your Sourcery config file:

metrics:
    enabled: True

New Refactorings

Enumerate

Loop counters that are incremented on each iteration of the loop like so:

i = 0
for animal in animals:
    i += 1
    print(i, animal)

can be replaced with a call to enumerate with a suitable start index:

for i, animal in enumerate(animals, start=1):
    print(i, animal)

Boolean IfExp identity

The following simple but overly verbose code:

return True if some_boolean_expression else False

is refactored to:

return bool(some_boolean_expression)

The negated version:

return False if some_boolean_expression else True

is refactored to:

return not some_boolean_expression

Writing a code editor plugin documentation

Instructions for how to write a code editor plugin are now available here. As the Sourcery binary implements the Language Server Protocol it is very easy to write a plugin for a new code editor.

Website changes

• GitHub repo page allows you to select which branch to create refactor PR for
• GitHub repo page shows instructions for starting refactoring reviews if none exist
• GitHub repo page shows all refactoring jobs, including running and failed ones

Free for 3 team members on private repos

We wanted to make it easier for you to bring Sourcery into your workplace so you can improve your codebase. Starting today you can now use Sourcery for free with up to 3 team members! This is for both public and private repos.

Sourcery will review and refactor all new pull requests for these team members and give you insight to code quality. The only limitation is that refactoring whole private repos is still disabled on the free tier.

Public repos, as always, get access to all of Sourcery's features for free.

Choosing team members

One of the big takeaways from some of our conversations with you is that teams != full GitHub organisations.

Now you can pick and choose individual team members within your organisation who should have Sourcery access.

All organisation members still get full Sourcery functionality for public repositories.

More insights into code quality

Our code quality tools are now available outside of GitHub! Right now they're hidden by default in PyCharm and VS Code, but they're very easy to enable.

To set them up just add the following setting to your Sourcery config file:

metrics:
    enabled: True

We're still in the alpha stage of our code quality tools within IDEs, so we'd love to get your feedback and thoughts as we continue to make improvements.