Part 4: Make an nf-core module¶

In this fourth part of the Hello nf-core training course, we show you how to create an nf-core module by applying the key conventions that make modules portable and maintainable.

The nf-core project provides a command (nf-core modules create) that generates properly structured module templates automatically, similar to what we used for the workflow in Part 2. However, for teaching purposes, we're going to start by doing it manually: transforming the local cowpy module in your core-hello pipeline into an nf-core-style module step-by-step. After that, we'll show you how to use the template-based module creation to work more efficiently in the future.

Note

This section assumes you have completed Part 3: Use an nf-core module and have integrated the CAT_CAT module into your pipeline.

If you did not complete Part 3 or want to start fresh for this part, you can use the core-hello-part3 solution as your starting point. Run these commands from inside the hello-nf-core/ directory:

cp -r solutions/core-hello-part3 core-hello
cd core-hello

This gives you a pipeline with the CAT_CAT module already integrated.

1. Transform `cowpy` into an nf-core module¶

In this section, we'll apply nf-core conventions to the local cowpy module in your core-hello pipeline, transforming it into a module that follows community standards.

We'll apply the following nf-core conventions incrementally:

Update cowpy to use metadata tuples to propagate sample metadata through the workflow.
Centralize tool argument configuration with ext.args to increase module versatility while keeping the interface minimal.
Standardize output naming with ext.prefix to promote consistency.
Centralize the publishing configuration to promote consistency.

After each step, we'll run the pipeline to test that everything works as expected.

Working directory

Make sure you're in the core-hello directory (your pipeline root) for all the commands and file edits in this section.

cd core-hello

1.1. Update `cowpy` to use metadata tuples¶

In the current version of the core-hello pipeline, we're extracting the file from CAT_CAT's output tuple to pass to cowpy.

It would be better to have cowpy accept metadata tuples directly, allowing metadata to flow on through the workflow. To the end, we'll need to make the following changes:

Update the input and output definitions
Update the process call in the workflow
Update the emit block in the workflow

Once we've done all that, we'll run the pipeline to test that everything still works as before.

1.1.1. Update the input and output definitions¶

Let's get started! Open the cowpy.nf module file (under core-hello/modules/local/) and modify it to accept metadata tuples as shown below.

AfterBefore

core-hello/modules/local/cowpy.nf
#!/usr/bin/env nextflow

// Generate ASCII art with cowpy (https://github.com/jeffbuttars/cowpy)
process cowpy {

    publishDir 'results', mode: 'copy'

    container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273'
    conda 'conda-forge::cowpy==1.1.5'

    input:
        tuple val(meta), path(input_file)
        val character

    output:
        tuple val(meta), path("cowpy-${input_file}"), emit: cowpy_output

    script:
    """
    cat $input_file | cowpy -c "$character" > cowpy-${input_file}
    """
}

core-hello/modules/local/cowpy.nf
#!/usr/bin/env nextflow

// Generate ASCII art with cowpy (https://github.com/jeffbuttars/cowpy)
process cowpy {

    publishDir 'results', mode: 'copy'

    container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273'
    conda 'conda-forge::cowpy==1.1.5'

    input:
        path input_file
        val character

    output:
        path "cowpy-${input_file}"

    script:
    """
    cat $input_file | cowpy -c "$character" > cowpy-${input_file}
    """
}

As you can see, we changed both the main input and the output to a tuple that follows the tuple val(meta), path(input_file) pattern introduced in Part 3 of this training. For the output, we also took this opportunity to add emit: cowpy_output in order to give the output channel a descriptive name.

Now that we've changed what the process expects, we need to update what we provide to it in the process call.

1.1.2. Update the process call in the workflow¶

The good news is that this change will simplify the process call. Now that the output of CAT_CAT and the input of cowpy are the same 'shape', i.e. they both consist of a tuple val(meta), path(input_file) structure, we can simply connect them directly instead of having to extract the file explicitly from the output of the CAT_CAT process.

Open the hello.nf workflow file (under core-hello/workflows/) and update the call to cowpy as shown below.

AfterBefore

core-hello/workflows/hello.nf
    // generate ASCII art of the greetings with cowpy
    cowpy(CAT_CAT.out.file_out, params.character)

core-hello/workflows/hello.nf
    // extract the file from the tuple since cowpy doesn't use metadata yet
    ch_for_cowpy = CAT_CAT.out.file_out.map{ meta, file -> file }

    // generate ASCII art of the greetings with cowpy
    cowpy(ch_for_cowpy, params.character)

We now call cowpy on CAT_CAT.out.file_out directly.

As a result, we no longer need to construct the ch_for_cowpy channel, so that line (and its comment line) can be deleted entirely.

1.1.3. Update the emit block in the workflow¶

Since cowpy now emits a named output, cowpy_output, we can update the hello.nf workflow's emit: block to use that.

AfterBefore

core-hello/workflows/hello.nf
    emit:
    cowpy_hellos   = cowpy.out.cowpy_output
    versions       = ch_versions                 // channel: [ path(versions.yml) ]

core-hello/workflows/hello.nf
    emit:
    cowpy_hellos   = cowpy.out
    versions       = ch_versions                 // channel: [ path(versions.yml) ]

This is technically not required, but it's good practice to refer to named outputs whenever possible.

1.1.4. Run the pipeline to test it¶

Let's run the workflow to test that everything is working correctly after these changes.

nextflow run . --outdir core-hello-results -profile test,docker --validate_params false

The pipeline should run successfully, with metadata now flowing from CAT_CAT through cowpy:

Output (excerpt)

executor >  local (8)
[b2/4cf633] CORE_HELLO:HELLO:sayHello (2)       [100%] 3 of 3 ✔
[ed/ef4d69] CORE_HELLO:HELLO:convertToUpper (3) [100%] 3 of 3 ✔
[2d/32c93e] CORE_HELLO:HELLO:CAT_CAT (test)     [100%] 1 of 1 ✔
[da/6f3246] CORE_HELLO:HELLO:cowpy              [100%] 1 of 1 ✔
-[core/hello] Pipeline completed successfully-

That completes what we needed to do to make cowpy handle metadata tuples. Now, let's look at what else we can do to take advantage of nf-core module patterns.

1.2. Centralize tool argument configuration with `ext.args`¶

In its current state, the cowpy process expects to receive a value for the character parameter. As a result, we have to provide a value every time we call the process, even if we'd be happy with the defaults set by the tool. For cowpy this is admittedly not a big problem, but for tools with many optional parameters, it can get quite cumbersome.

The nf-core project recommends using a Nextflow feature called ext.args to manage tool arguments more conveniently via configuration files.

Instead of declaring process inputs for every tool option, you write the module to reference ext.args in the construction of its command line. Then it's just a matter of setting up the ext.args variable to hold the arguments and values you want to use in the modules.config file, which consolidates configuration details for all modules. Nextflow will add those arguments with their values into the tool command line at runtime.

Let's apply this approach to the cowpy module. We're going to need to make the following changes:

Update the cowpy module
Configure ext.args in the modules.config file
Update the hello.nf workflow

Once we've done all that, we'll run the pipeline to test that everything still works as before.

1.2.1. Update the `cowpy` module¶

Let's do it. Open the cowpy.nf module file (under core-hello/modules/local/) and modify it to reference ext.args as shown below.

AfterBefore

modules/local/cowpy.nf
#!/usr/bin/env nextflow

// Generate ASCII art with cowpy (https://github.com/jeffbuttars/cowpy)
process cowpy {

    publishDir 'results', mode: 'copy'

    container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273'
    conda 'conda-forge::cowpy==1.1.5'

    input:
        tuple val(meta), path(input_file)

    output:
        tuple val(meta), path("cowpy-${input_file}"), emit: cowpy_output

    script:
    def args = task.ext.args ?: ''
    """
    cat $input_file | cowpy $args > cowpy-${input_file}
    """
}

core-hello/modules/local/cowpy.nf
#!/usr/bin/env nextflow

// Generate ASCII art with cowpy (https://github.com/jeffbuttars/cowpy)
process cowpy {

    publishDir 'results', mode: 'copy'

    container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273'
    conda 'conda-forge::cowpy==1.1.5'

    input:
        tuple val(meta), path(input_file)
        val character

    output:
        tuple val(meta), path("cowpy-${input_file}"), emit: cowpy_output

    script:
    """
    cat $input_file | cowpy -c "$character" > cowpy-${input_file}
    """
}

You can see we made three changes.

In the input: block, we removed the val character input. Going forward, we'll supply that argument via the ext.args configuration as described further below.
In the script: block, we added the line def args = task.ext.args ?: ''. That line uses the ?: operator to determine the value of the args variable: the content of task.ext.args if it is not empty, or an empty string if it is. Note that while we generally refer to ext.args, this code must reference task.ext.args to pull out the module-level ext.args configuration.
In the command line, we replaced -c "$character" with $args. This is where Nextflow will inject any tool arguments set in ext.args in the modules.config file.

As a result, the module interface is now simpler: it only expects the essential metadata and file inputs.

Note

The ?: operator is often called the 'Elvis operator' because it looks like a sideways Elvis Presley face, with the ? character symbolizing the wave in his hair.

1.2.2. Configure `ext.args` in the `modules.config` file¶

Now that we've taken the character declaration out of the module, we've got to add it to ext.args in the modules.config configuration file.

Specifically, we're going to add this little chunk of code to the process {} block:

Code to add

withName: 'cowpy' {
    ext.args = { "-c ${params.character}" }
}

The withName: syntax assigns this configuration to the cowpy process only, and ext.args = { "-c ${params.character}" } simply composes a string that will include the value of the character parameter. Note the use of curly braces, which tell Nextflow to evaluate the value of the parameter at runtime.

Makes sense? Let's add it in.

Open conf/modules.config and add the configuration code inside the process {} block as shown below.

AfterBefore

core-hello/conf/modules.config
process {
    publishDir = [
        path: { "${params.outdir}/${task.process.tokenize(':')[-1].tokenize('_')[0].toLowerCase()}" },
    ]

    withName: 'cowpy' {
        ext.args = { "-c ${params.character}" }
    }
}

core-hello/conf/modules.config
process {
    publishDir = [
        path: { "${params.outdir}/${task.process.tokenize(':')[-1].tokenize('_')[0].toLowerCase()}" },
    ]
}

Hopefully you can imagine having all the modules in a pipeline have their ext.args specified in this file, with the following benefits:

The module interface stays simple - It only accepts the essential metadata and file inputs
The pipeline still exposes params.character - End-users can still configure it as before
The module is now portable - It can be reused in other pipelines without expecting a specific parameter name
The configuration is centralized in modules.config, keeping workflow logic clean

By using the modules.config file as the place where all pipelines centralize per-module configuration, we make our modules more reusable across different pipelines.

1.2.3. Update the `hello.nf` workflow¶

Since the cowpy module no longer requires the character parameter as an input, we need to update the workflow call accordingly.

Open the hello.nf workflow file (under core-hello/workflows/) and update the call to cowpy as shown below.

AfterBefore

core-hello/workflows/hello.nf
    // generate ASCII art of the greetings with cowpy
    cowpy(CAT_CAT.out.file_out)

core-hello/workflows/hello.nf
    // generate ASCII art of the greetings with cowpy
    cowpy(CAT_CAT.out.file_out, params.character)

The workflow code is now cleaner: we don't need to pass params.character directly to the process. The module interface is kept minimal, making it more portable, while the pipeline still provides the explicit option through configuration.

1.2.4. Run the pipeline to test it¶

Let's test that the workflow still works as expected, specifying a different character to verify that the ext.args configuration is working.

Run this command using kosh, one of the more... enigmatic options:

nextflow run . --outdir core-hello-results -profile test,docker --validate_params false --character kosh

The pipeline should run successfully. In the output, look for the cowpy process execution line, which will show something like this:

Output (excerpt)

[bd/0abaf8] CORE_HELLO:HELLO:cowpy              [100%] 1 of 1 ✔

So it ran successfully, great! Now let's verify that the ext.args configuration worked by checking the output. Find the output in the file browser or use the task hash (the bd/0abaf8 part in the example above) to look at the output file:

cat work/bd/0abaf8*/cowpy-test.txt

Output

_________
/ HELLO   \
| HOLà    |
\ BONJOUR /
---------
    \
    \
      \
  ___       _____     ___
/   \     /    /|   /   \
|     |   /    / |  |     |
|     |  /____/  |  |     |
|     |  |    |  |  |     |
|     |  | {} | /   |     |
|     |  |____|/    |     |
|     |    |==|     |     |
|      \___________/      |
|                         |
|                         |

You should see the ASCII art displayed with the kosh character, confirming that the ext.args configuration worked!

Optional: Inspect the command file

If you want to see exactly how the configuration was applied, you can inspect the .command.sh file:

cat work/bd/0abaf8*/.command.sh

You'll see the cowpy command with the -c kosh argument:

#!/usr/bin/env bash
...
cat test.txt | cowpy -c kosh > cowpy-test.txt

This shows that the .command.sh file was generated correctly based on the ext.args configuration.

Take a moment to think about what we achieved here. This approach keeps the module interface focused on essential data (files, metadata, and any mandatory per-sample parameters), while options that control the behavior of the tool are handled separately through configuration.

This may seem unnecessary for a simple tool like cowpy, but it can make a big difference for data analysis tools that have a lot of optional arguments.

To summarize the benefits of this approach:

Clean interface: The module focuses on essential data inputs (metadata and files)
Flexibility: Users can specify tool arguments via configuration, including sample-specific values
Consistency: All nf-core modules follow this pattern
Portability: Modules can be reused without hardcoded tool options
No workflow changes: Adding or changing tool options doesn't require updating workflow code

Note

The ext.args system has powerful additional capabilities not covered here, including switching argument values dynamically based on metadata. See the nf-core module specifications for more details.

1.3. Standardize output naming with `ext.prefix`¶

Now that we've given the cowpy process access to the metamap, we can start taking advantage of another useful nf-core pattern: naming output files based on metadata.

Here we're going to use a Nextflow feature called ext.prefix that will allow us to standardize output file naming across modules using meta.id (the identifier included in the metamap), while still being able to configure modules individually if desired.

This will be similar to what we did with ext.args, with a few differences that we'll detail as we go.

Let's apply this approach to the cowpy module. We're going to need to make the following changes:

Update the cowpy module
Configure ext.prefix in the modules.config file

(No changes need to the workflow.)

Once we've done that, we'll run the pipeline to test that everything still works as before.

1.3.1. Update the `cowpy` module¶

Let's do it. Open the cowpy.nf module file (under core-hello/modules/local/) and modify it to reference ext.prefix as shown below.

AfterBefore

core-hello/modules/local/cowpy.nf
#!/usr/bin/env nextflow

// Generate ASCII art with cowpy (https://github.com/jeffbuttars/cowpy)
process cowpy {

    publishDir 'results', mode: 'copy'

    container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273'
    conda 'conda-forge::cowpy==1.1.5'

    input:
        tuple val(meta), path(input_file)

    output:
        tuple val(meta), path("${prefix}.txt"), emit: cowpy_output

    script:
    def args = task.ext.args ?: ''
    prefix = task.ext.prefix ?: "${meta.id}"
    """
    cat $input_file | cowpy $args > ${prefix}.txt
    """
}

core-hello/modules/local/cowpy.nf
#!/usr/bin/env nextflow

// Generate ASCII art with cowpy (https://github.com/jeffbuttars/cowpy)
process cowpy {

    publishDir 'results', mode: 'copy'

    container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273'
    conda 'conda-forge::cowpy==1.1.5'

    input:
        tuple val(meta), path(input_file)

    output:
        tuple val(meta), path("cowpy-${input_file}"), emit: cowpy_output

    script:
    def args = task.ext.args ?: ''
    """
    cat $input_file | cowpy $args > cowpy-${input_file}
    """
}

You can see we made three changes.

In the script: block, we added the line prefix = task.ext.prefix ?: "${meta.id}". That line uses the ?: operator to determine the value of the prefix variable: the content of task.ext.prefix if it is not empty, or the identifier from the metamap (meta.id) if it is. Note that while we generally refer to ext.prefix, this code must reference task.ext.prefix to pull out the module-level ext.prefix configuration.
In the command line, we replaced cowpy-${input_file} with ${prefix}.txt. This is where Nextflow will inject the value of prefix determined by the line above.
In the output: block, we replaced path("cowpy-${input_file}") with path("${prefix}.txt").** This simply reiterates what the file path will be according to what is written in the command line.

As a result, the output file name is now constructed using a sensible default (the identifier from the metamap) combined with the appropriate file format extension.

1.3.2. Configure `ext.prefix` in the `modules.config` file¶

In this case the sensible default is not sufficiently expressive for our taste; we want to use a custom naming pattern that includes the tool name, cowpy-<id>.txt, like we had before.

We'll do that by configuring ext.prefix in modules.config, just like we did for the character parameter with ext.args, except this time the withName: 'cowpy' {} block already exists, and we just need to add the following line:

Code to add

ext.prefix = { "cowpy-${meta.id}" }

This will compose the string we want. Note that once again we use curly braces, this time to tell Nextflow to evaluate the value of meta.id at runtime.

Let's add it in.

Open conf/modules.config and add the configuration code inside the process {} block as shown below.

AfterBefore

core-hello/conf/modules.config
    withName: 'cowpy' {
        ext.args = { "-c ${params.character}" }
        ext.prefix = { "cowpy-${meta.id}" }
    }

core-hello/conf/modules.config
    withName: 'cowpy' {
        ext.args = { "-c ${params.character}" }
    }

In case you're wondering, the ext.prefix closure has access to the correct piece of metadata because the configuration is evaluated in the context of the process execution, where metadata is available.

1.3.3. Run the pipeline to test it¶

Let's test that the workflow still works as expected.

nextflow run . --outdir core-hello-results -profile test,docker --validate_params false

Check the outputs:

ls results/

You should see the cowpy output file with the same naming as before: cowpy-test.txt, based on the default batch name. Feel free to change the ext.prefix configuration to satisfy yourself that you can change the naming pattern without having to make any changes to the module or workflow code.

Alternatively, you can also try running this again with a different --batch parameter specified on the command line to satisfy yourself that that part is still customizable on the fly.

This demonstrates how ext.prefix allows you to maintain your preferred naming convention while keeping the module interface flexible.

To summarize the benefits of this approach:

Standardized naming: Output files are typically named using sample IDs from metadata
Configurable: Users can override the default naming if needed
Consistent: All nf-core modules follow this pattern
Predictable: Easy to know what output files will be called

Pretty good, right? Well, there's one more important change we need to make to improve our module to fit the nf-core guidelines.

1.4. Centralize the publishing configuration¶

You may have noticed that we've been publishing outputs to two different directories:

results — The original output directory we've been using from the beginning for our local modules, set individually using per-module publishDir directives;
core-hello-results — The output directory set with --outdir on the command line, which has been receiving the nf-core logs and the results published by CAT_CAT.

This is messy and suboptimal; it would be better to have one location for everything. Of course, we could go into each of our local modules and update the publishDir directive manually to use the core-hello-results directory, but what about next time we decide to change the output directory?

Having individual modules make publishing decisions is clearly not the way to go, especially in a world where the same module might be used in a lot of different pipelines, by people who have different needs or preferences. We want to be able to control where outputs get published at the level of the workflow configuration.

"Hey," you might say, "CAT_CAT is sending its outputs to the --outdir. Maybe we should copy its publishDir directive?"

Yes, that's a great idea.

Except it doesn't have a publishDir directive. (Go ahead, look at the module code.)

That's because nf-core pipelines centralize control at the workflow level by configuring publishDir in conf/modules.config rather than in individual modules. Specifically, the nf-core template declares a default publishDir directive (with a predefined directory structure) that applies to all modules unless an overriding directive is provide.

Doesn't that sound awesome? Could it be that to take advantage of this default directive, all we need to do is remove the current publishDir directive from our local modules?

Let's try that out on cowpy to see what happens, then we'll look at the code for the default configuration to understand how it works.

Finally, we'll demonstrate how to override the default behavior if desired.

1.4.1. Remove the `publishDir` directive from `cowpy`¶

Let's do this. Open the cowpy.nf module file (under core-hello/modules/local/) and remove the publishDir directive as shown below.

AfterBefore

core-hello/modules/local/cowpy.nf (excerpt)
#!/usr/bin/env nextflow

// Generate ASCII art with cowpy (https://github.com/jeffbuttars/cowpy)
process cowpy {

    container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273'
    conda 'conda-forge::cowpy==1.1.5'

core-hello/modules/local/cowpy.nf (excerpt)
#!/usr/bin/env nextflow

// Generate ASCII art with cowpy (https://github.com/jeffbuttars/cowpy)
process cowpy {

    publishDir 'results', mode: 'copy'

    container 'community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273'
    conda 'conda-forge::cowpy==1.1.5'

That's it!

1.4.2. Run the pipeline to see what happens¶

Let's have a look at what happens if we run the pipeline now.

nextflow run . --outdir core-hello-results -profile test,docker --validate_params false

Have a look at your current working directory. Now the core-hello-results also contains the outputs of the cowpy module.

tree core-hello-results/

You can see that Nextflow created this hierarchy of directories based on the names of the workflow and of the module.

The code responsible lives in the conf/modules.config file. This is the default publishDir configuration that is part of the nf-core template and applies to all processes.

process {
    publishDir = [
        path: { "${params.outdir}/${task.process.tokenize(':')[-1].tokenize('_')[0].toLowerCase()}" },
        mode: params.publish_dir_mode,
        saveAs: { filename -> filename.equals('versions.yml') ? null : filename }
    ]
}

This may look complicated, so let's look at each of the three components:

path: Determines the output directory based on the process name. The full name of a process contained in task.process includes the hierarchy of workflow and module imports (such as CORE_HELLO:HELLO:CAT_CAT). The tokenize operations strip away that hierarchy to get just the process name, then take the first part before any underscore (if applicable), and convert it to lowercase. This is what determines that the results of CAT_CAT get published to ${params.outdir}/cat/.
mode: Controls how files are published (copy, symlink, etc.). This is configurable via the params.publish_dir_mode parameter.
saveAs: Filters which files to publish. This example excludes versions.yml files by returning null for them, preventing them from being published.

This provides a consistent logic for organizing outputs.

The output looks even better when all the modules in a pipeline adopt this convention, so feel free to go delete the publishDir directives from the other modules in your pipeline. This default will be applied even to modules that we didn't explicitly modify to follow nf-core guidelines.

That being said, you may decide you want to organize your inputs differently, and the good news is that it's easy to do so.

1.4.3. Override the default¶

To override the default publishDir directive, you can simply add your own directives to the conf/modules.config file.

For example, you could override the default for a single process using the withName: selector, as in this example where we add a custom publishDir directive for the 'cowpy' process.

core-hello/conf/modules.config
process {
    publishDir = [
        path: { "${params.outdir}/${task.process.tokenize(':')[-1].tokenize('_')[0].toLowerCase()}" },
    ]

    withName: 'cowpy' {
        ext.args = { "-c ${params.character}" }
        publishDir = [
            path: 'my_custom_results'
        ]
    }
}

We're not actually going to make that change, but feel free to play with this and see what logic you can implement.

The point is that this system allows gives you the best of both worlds: consistency by default and the flexibility to customize the configuration on demand.

To summarize, you get:

Single source of truth: All publishing configuration lives in modules.config
Useful default: Processes work out-of-the-box without per-module configuration
Easy customization: Override publishing behavior in config, not in module code
Portable modules: Modules don't hardcode output locations

This completes the set of nf-core module features you should absolutely learn to use, but there are others which you can read about in the nf-core modules specifications.

Takeaway¶

You now know how to adapt local modules to follow nf-core conventions:

Design your modules to accept and propagate metadata tuples;
Use ext.args to keep module interfaces minimal and portable;
Use ext.prefix for configurable, standardized output file naming;
Adopt the default centralized publishDir directive for a consistent results directory structure.

What's next?¶

Learn how to use nf-core's built-in template-based tools to create modules the easy way.

2. Generate modules with nf-core tools¶

Now that you've learned the nf-core module patterns by applying them manually, let's look at how you'd create modules in practice. The nf-core project provides the nf-core modules create command that generates properly structured module templates with all these patterns built in from the start.

2.1. Using nf-core modules create¶

The nf-core modules create command generates a module template that already follows all the conventions you've learned.

For example, to create the cowpy module with a minimal template:

nf-core modules create --empty-template cowpy

The --empty-template flag creates a clean starter template without extra code, making it easier to see the essential structure.

The command runs interactively, guiding you through the setup. It automatically looks up tool information from package repositories like Bioconda and bio.tools to pre-populate metadata.

You'll be prompted for several configuration options:

Author information: Your GitHub username for attribution
Resource label: A predefined set of computational requirements. The nf-core project provides standard labels like process_single for lightweight tools and process_high for demanding ones. These labels help manage resource allocation across different execution environments.
Metadata requirement: Whether the module needs sample-specific information via a meta map (usually yes for data processing modules).

The tool handles the complexity of finding package information and setting up the structure, allowing you to focus on implementing the tool's specific logic.

2.2. What gets generated¶

The tool creates a complete module structure in modules/local/ (or modules/nf-core/ if you're in the nf-core/modules repository):

Directory contents

modules/local/cowpy
├── environment.yml
├── main.nf
├── meta.yml
└── tests
    └── main.nf.test

Each file serves a specific purpose:

main.nf: Process definition with all the nf-core patterns built in
meta.yml: Module documentation describing inputs, outputs, and the tool
environment.yml: Conda environment specification for dependencies
tests/main.nf.test: nf-test test cases to validate the module works

Learn more about testing

The generated test file uses nf-test, a testing framework for Nextflow pipelines and modules. To learn how to write and run these tests, see the nf-test side quest.

The generated main.nf includes all the patterns you just learned, plus some additional features:

modules/local/cowpy/main.nf

process COWPY {
    tag "$meta.id"
    label 'process_single'

    conda "${moduleDir}/environment.yml"
    container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
        'https://depot.galaxyproject.org/singularity/YOUR-TOOL-HERE':
        'biocontainers/YOUR-TOOL-HERE' }"

    input:
    tuple val(meta), path(input)        // Pattern 1: Metadata tuples ✓

    output:
    tuple val(meta), path("*"), emit: output
    path "versions.yml"           , emit: versions

    when:
    task.ext.when == null || task.ext.when

    script:
    def args = task.ext.args ?: ''              // Pattern 2: ext.args ✓
    def prefix = task.ext.prefix ?: "${meta.id}"  // Pattern 3: ext.prefix ✓

    """
    // Add your tool command here

    cat <<-END_VERSIONS > versions.yml
    "${task.process}":
        cowpy: \$(cowpy --version)
    END_VERSIONS
    """

    stub:
    def args = task.ext.args ?: ''
    def prefix = task.ext.prefix ?: "${meta.id}"

    """
    echo $args
    touch ${prefix}.txt

    cat <<-END_VERSIONS > versions.yml
    "${task.process}":
        cowpy: \$(cowpy --version)
    END_VERSIONS
    """
}

Notice how all the patterns you applied manually above are already there! The template also includes several additional nf-core conventions. Some of these work out of the box, while others are placeholders we'll need to fill in, as described below.

Features that work as-is:

tag "$meta.id": Adds sample ID to process names in logs for easier tracking
label 'process_single': Resource label for configuring CPU/memory requirements
when: block: Allows conditional execution via task.ext.when configuration

These features are already functional and make modules more maintainable.

Placeholders we'll customize below:

input: and output: blocks: Generic declarations we'll update to match our tool
script: block: Contains a comment where we'll add the cowpy command
stub: block: Template we'll update to produce the correct outputs
Container and environment: Placeholders we'll fill with package information

The next sections walk through completing these customizations.

2.3. Completing the environment and container setup¶

In the case of cowpy, the tool warned that it couldn't find the package in Bioconda (the primary channel for bioinformatics tools). However, cowpy is available in conda-forge, so you would complete the environment.yml like this:

modules/local/cowpy/environment.yml

name: cowpy
channels:
  - conda-forge
dependencies:
  - cowpy=1.1.5

For the container, you can use Seqera Containers to automatically build a container from any Conda package, including conda-forge packages:

container "community.wave.seqera.io/library/cowpy:1.1.5--3db457ae1977a273"

Bioconda vs conda-forge packages

Bioconda packages: Automatically get BioContainers built, providing ready-to-use containers
conda-forge packages: Can use Seqera Containers to build containers on-demand from the Conda recipe

Most bioinformatics tools are in Bioconda, but for conda-forge tools, Seqera Containers provides an easy solution for containerization.

2.4. Defining inputs and outputs¶

The generated template includes generic input and output declarations that you'll need to customize for your specific tool. Looking back at our manual cowpy module from section 1, we can use that as a guide.

Update the input and output blocks:

AfterBefore

modules/local/cowpy/main.nf
input:
tuple val(meta), path(input_file)

output:
tuple val(meta), path("${prefix}.txt"), emit: cowpy_output
path "versions.yml"           , emit: versions

modules/local/cowpy/main.nf
input:
tuple val(meta), path(input)

output:
tuple val(meta), path("*"), emit: output
path "versions.yml"           , emit: versions

This specifies:

The input file parameter name (input_file instead of generic input)
The output filename using the configurable prefix pattern (${prefix}.txt instead of wildcard *)
A descriptive emit name (cowpy_output instead of generic output)

2.5. Writing the script block¶

The template provides a comment placeholder where you add the actual tool command. We can reference our manual module from earlier for the command logic:

AfterBefore

modules/local/cowpy/main.nf
script:
def args = task.ext.args ?: ''
prefix = task.ext.prefix ?: "${meta.id}"

"""
cat $input_file | cowpy $args > ${prefix}.txt

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    cowpy: \$(cowpy --version)
END_VERSIONS
"""

modules/local/cowpy/main.nf
script:
def args = task.ext.args ?: ''
def prefix = task.ext.prefix ?: "${meta.id}"

"""
// Add your tool command here

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    cowpy: \$(cowpy --version)
END_VERSIONS
"""

Key changes:

Change def prefix to just prefix (without def) so it's accessible in the output block
Replace the comment with the actual cowpy command that uses both $args and ${prefix}.txt

2.6. Implementing the stub block¶

The stub block provides a fast mock implementation for testing pipeline logic without running the actual tool. It must produce the same output files as the script block:

AfterBefore

modules/local/cowpy/main.nf
stub:
def args = task.ext.args ?: ''
prefix = task.ext.prefix ?: "${meta.id}"

"""
touch ${prefix}.txt

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    cowpy: \$(cowpy --version)
END_VERSIONS
"""

modules/local/cowpy/main.nf
stub:
def args = task.ext.args ?: ''
def prefix = task.ext.prefix ?: "${meta.id}"

"""
echo $args
touch ${prefix}.txt

cat <<-END_VERSIONS > versions.yml
"${task.process}":
    cowpy: \$(cowpy --version)
END_VERSIONS
"""

Key changes:

Change def prefix to just prefix to match the script block
Remove the echo $args line (which was just template placeholder code)
The stub creates an empty ${prefix}.txt file matching what the script block produces

This allows you to test workflow logic and file handling without waiting for the actual tool to run.

Once you've completed the environment setup (section 2.1.2), inputs/outputs (section 2.1.3), script block (section 2.1.4), and stub block (section 2.1.5), the module is ready to test!

Takeaway¶

You now know how to use the built-in nf-core tooling to create modules efficiently using templates rather than writing everything from scratch.

What's next?¶

Learn what are the benefits of contributing modules to nf-core and what are the main steps and requirements involved.

3. Contributing modules back to nf-core¶

The nf-core/modules repository welcomes contributions of well-tested, standardized modules.

3.1. Why contribute?¶

Contributing your modules to nf-core:

Makes your tools available to the entire nf-core community through the modules catalog at nf-co.re/modules
Ensures ongoing community maintenance and improvements
Provides quality assurance through code review and automated testing
Gives your work visibility and recognition

3.2. Contributor's checklist¶

To contribute a module to nf-core, you will need to go through the following steps:

Check if it already exists at nf-co.re/modules
Fork the nf-core/modules repository
Use nf-core modules create to generate the template
Fill in the module logic and tests
Test with nf-core modules test tool/subtool
Lint with nf-core modules lint tool/subtool
Submit a pull request

For detailed instructions, see the nf-core components tutorial.

3.3. Resources¶

Components tutorial: Complete guide to creating and contributing modules
Module specifications: Technical requirements and guidelines
Community support: nf-core Slack - Join the #modules channel

Takeaway¶

You now know how to create nf-core modules! You learned the four key patterns that make modules portable and maintainable:

Metadata tuples propagate metadata through the workflow
ext.args simplifies module interfaces by handling optional arguments via configuration
ext.prefix standardizes output file naming
Centralized publishing via publishDir configured in modules.config rather than hardcoded in modules

By transforming cowpy step-by-step, you developed a deep understanding of these patterns, making you equipped to work with, debug, and create nf-core modules. In practice, you'll use nf-core modules create to generate properly structured modules with these patterns built in from the start.

Finally, you learned how to contribute modules to the nf-core community, making tools available to researchers worldwide while benefiting from ongoing community maintenance.

What's next?¶

When you're ready, continue to Part 5: Input validation to learn how to add schema-based input validation to your pipeline.

Part 4: Make an nf-core module¶

1. Transform cowpy into an nf-core module¶

1.1. Update cowpy to use metadata tuples¶

1.1.1. Update the input and output definitions¶

1.1.2. Update the process call in the workflow¶

1.1.3. Update the emit block in the workflow¶

1.1.4. Run the pipeline to test it¶

1.2. Centralize tool argument configuration with ext.args¶

1.2.1. Update the cowpy module¶

1.2.2. Configure ext.args in the modules.config file¶

1.2.3. Update the hello.nf workflow¶

1.2.4. Run the pipeline to test it¶

1.3. Standardize output naming with ext.prefix¶

1.3.1. Update the cowpy module¶

1.3.2. Configure ext.prefix in the modules.config file¶

1.3.3. Run the pipeline to test it¶

1.4. Centralize the publishing configuration¶

1.4.1. Remove the publishDir directive from cowpy¶

1.4.2. Run the pipeline to see what happens¶

1.4.3. Override the default¶

Takeaway¶

What's next?¶

2. Generate modules with nf-core tools¶

2.1. Using nf-core modules create¶

2.2. What gets generated¶

2.3. Completing the environment and container setup¶

2.4. Defining inputs and outputs¶

2.5. Writing the script block¶

2.6. Implementing the stub block¶

Takeaway¶

What's next?¶

3. Contributing modules back to nf-core¶

3.1. Why contribute?¶

3.2. Contributor's checklist¶

3.3. Resources¶

Takeaway¶

What's next?¶

1. Transform `cowpy` into an nf-core module¶

1.1. Update `cowpy` to use metadata tuples¶

1.2. Centralize tool argument configuration with `ext.args`¶

1.2.1. Update the `cowpy` module¶

1.2.2. Configure `ext.args` in the `modules.config` file¶

1.2.3. Update the `hello.nf` workflow¶

1.3. Standardize output naming with `ext.prefix`¶

1.3.1. Update the `cowpy` module¶

1.3.2. Configure `ext.prefix` in the `modules.config` file¶

1.4.1. Remove the `publishDir` directive from `cowpy`¶