For any Salesforce developer, knowing how to retrieve metadata is an essential skill that will enhance your ability to manage, deploy, and scale Salesforce configurations effectively. This process supports robust version control, reduces errors made from manual changes, enhances collaborative development, and plays a critical role in CI/CD.

‍

In this guide, we’ll cover the ins and outs of retrieving Salesforce metadata, including overviews of the metadata API, the tooling API, and retrieving commands from the SFDX CLI. For our purposes here, we’ll be interacting with the metadata API through Salesforce DX, but you could also create your own applications with the SOAP and REST APIs if you prefer.

‍

‍

The Metadata API

A Brief Intro

Imagine that you’re deploying a flow. You’re having some issues, and you want to understand how the flow metadata is modeled in Salesforce. If you use Google to search for “flow metadata API Salesforce”, the first result will usually be that metadata type in the Metadata API Developer Guide—which is a resource you’ll want to keep handy.

‍

‍

In the Developer Guide, we see all the different attributes a flow can have, such as assignments, choices, decisions, etc. It may also show us an example of what the XML would look like. When we look at Salesforce metadata in a repository or an SFDX project, anything that is not code (Apex Lightning web components, VisualForce, etc.) will be stored as XML, and that XML representation comes from the metadata API—which is why it’s important to be aware of it.

‍

There are three main components to the metadata API that you need to know. The first thing to know is that there are two main API calls: the retrieve and the deploy calls, which, as the name implies, are all about retrieving and deploying metadata to a Salesforce org.

‍

The other thing you have to be aware of is that these two calls rely on a package.xml, which is an XML file that has the names of the metadata in your org. For example, in the image below you can see the name of a custom object with the name of a profile. This is used by these two calls that deploy and retrieve from your Salesforce org. 

‍

Quick Start Overview

The metadata API is huge, and there’s no way we can cover all of it here. However, in case you haven’t used the metadata API before, let’s take a quick look at the Quick Start section of the Developer Guide so you can get a feel for things. If you’re already familiar with the metadata API, feel free to skip this section.

‍

The first part of the Quick Start explains how the package.xml works, and the second part shows us how to work with it. Let’s follow those steps now.

‍

First, we’ll create an SFDX project. Then, we’ll create a new directory called “metadata_api” and go to that directory, making sure to copy the command shown in the Quick Start. When you paste that command, you’ll replace “YourProjectName” with whatever name you’ve decided to give your new project. We’ll use “sample_metadata” for our example. This will create a new Salesforce DX project with that name.

‍

Heading to that folder, we’ll then open the project in Visual Studio Code. Since this is an empty SFDX project, we’ll then link it to one of our orgs.

‍

‍

Back in the Quick Start guide, you’ll see that it gives us a script with which to create a package.xml, so we’ll copy that script back to our new SFDX project and name it “package.xml”. We’ll also change it to specify that we want to retrieve the account object. This might look weird because the account is not a custom object, but if we try to retrieve an account using “object” or something like “standard object”, it’s not going to work. This is another reason it’s important to know the metadata API. We’ll also change it to say that we want to retrieve all the custom fields of the account object, then hit save.

‍

‍

‍

Going back to the Quick Start guide, we’ll see that it asks us to run an SFDX command using the retrieve command. This is because the metadata API is baked into the Salesforce CLI. So, by using this command, we’re calling the API behind the scenes. Go ahead and copy that whole command.

‍

‍

Back in the project, let’s issue that retrieve request. Once we’ve retrieved all the metadata, we can open the objects folder and see all the fields of the account object. Open those, and we’ll see the attributes in XML, which follows the format of the metadata API.

‍

‍

The Tooling API

A Brief Intro

Like the metadata API, the tooling API is also SOAP and REST based and can be used to retrieve and deploy metadata to a Salesforce org. The main difference between the metadata and tooling APIs is that the tooling API provides fine grained access to specific attributes of the metadata. We’ll look at an example soon.

‍

The tooling API can also be accessed through the Salesforce DX CLI and is often less relevant to CI/CD in the sense that it’s not really used for retrieving and deploying metadata. However, you’ll still see it used in SFDX plugins that may need to retrieve additional metadata not commonly available with the metadata API.

‍

Unlike with the metadata API, the documentation of the tooling API is a bit raw; it can be hard to understand the real use cases or how to actually use it. For example, if we navigate to the REST overview section of the documentation, we’ll see some examples, but they can be tricky to really “get”, so let’s take a firsthand look at the tooling API in a Salesforce org.

The Tooling API in Action

The best way to look at the tooling API is to actually go to a custom field in our org. We’ll use an extension called the Salesforce Inspector (which we really recommend, by the way) to look at the data of this custom field using the tooling API. Click on “Show all data”.

‍

‍

A new window will open, and we’ll see every single attribute of that field as represented by the tooling API, such as the field name, the custom field definition ID, and the full metadata name. When we scroll down, we see a huge metadata attribute, which is a big JSON string with a lot more details about this field, such as the different values of this pick list. This provides much more information than what you would normally see with the metadata API.

‍

‍

One thing that is very different from the metadata API is that rather than having just a retrieve call, the tooling API lets you use normal SOQL queries to retrieve individual attributes of the metadata (like those shown above).

‍

We can use the Salesforce DX CLI and run a query against a tooling API to retrieve metadata from a specific metadata item. Let’s look at an example.

‍

Going back to our custom field, we can see the ID in the top right. We’ll copy that, then head back to our SFDX project (which we created above) and drop that ID there. Then, we’ll write a query (see below) and put everything in a JSON format. We’ll also make sure to include -t for tooling API and send the output of this to a file (called myFieldDetails in this example).

‍

‍

Look for the myFieldDetails.json file in the left side menu, and we’ll then see the attributes of that custom field. This looks very different from what we saw earlier in the metadata API. If we look at the field that we just got that metadata from, we’ll see that in the metadata API format, the values we see are just the name and the value set definition, the actual values of the pick list. But we don’t see anything else about this field.

‍

‍

However, in the tooling API, when we pull the metadata object from the SOQL query, we get a lot more data—which is why it’s called fine grained access, because we can choose what to actually retrieve from this field.

‍

‍

Let’s say we want to see the ‘last modified by’ ID for this custom field. Let’s rewrite that query, and, instead of getting the metadata, let’s get the ‘last modified by’ ID. 

‍

‍

We should see our user ID in the org as we were the last user to edit the field. As we saw previously, this information is not available immediately on the metadata API. There’s a way to get it, but generally speaking when you’re retrieving using the metadata API using the Salesforce CLI (like we did earlier), you will get basic information and get it all in one go. You cannot ask the metadata API to give you only certain information—you’ll get the whole chunk. However, in the tooling API, you can write SOQL queries and be very specific about what you want to retrieve from that metadata

‍

Although the tooling API is not really used for CI/CD for the purpose of retrieving and deploying to another org, it can be used to help retrieve additional information that is not normally available via the metadata API. This can be very helpful when implementing a CI/CD solution, so let’s look at an example.

‍

The tooling API comes with this object called the Metadata Component Dependency, which is in beta and has been in beta for a few years. Basically, this object allows you to retrieve the dependencies of any metadata, so if you have a custom field and you want to see where that custom field is being used, you can use this API to get that information. You can query this object using normal SOQL queries.

‍

‍

Let’s say we have a field that we added to a bunch of other customization, like a page layout, a couple of flows, maybe some process builders, workflow rules, workflow field updates, etc. Now, we want to deploy all this metadata to another org, but we can’t remember all the places we added the field. This is a scenario where we could use the Metadata Component Dependency to get that information.

‍

We can use the same SOQL query format that we saw earlier on the tooling API, but the query is against the Metadata Component Dependency object. What we’re saying in this query is basically, “Get me all the dependencies linked to this ID”, and that is the ID of the custom field that we got from our Salesforce org. 

‍

‍

Once we hit enter, the tooling API is going to calculate all the places where this field is being used. We can see that there are a few Apex classes, a few formula fields, page layouts, and a few flows. 

‍

‍

This is great, but it’s not exactly what we need because we can’t retrieve this metadata as is. This is not a package.xml that we can retrieve using the Salesforce CLI. Also, if we look at the names of the fields or page layouts, we see they are not complete. For example, the Salesforce API name would usually be “account.account_priority” or “account.priority_formula”, or in the case of a page layout, it’d be “account-account layout”. This means that even if we had this in a package.xml format, the names would not be correct and we couldn’t use the metadata API to retrieve it. In this case, we can use a CLI plugin or any other application on top of this API to provide that added capability. 

‍

Just like the metadata API, this concept is huge. The main thing to remember is that the tooling API can be used to retrieve specific values from a piece of metadata and that it may be useful in certain CI/CD operations, such as dependency management.

Retrieving Metadata

Retrieving Salesforce metadata is important for CI/CD because the core of CI/CD is having your Salesforce metadata in a Git provider like Bitbucket, GitHub, etc. Additionally, the more metadata you track in Git, the more Git becomes the source of truth for changes that are happening in your Salesforce org.

‍

Until now, we’ve only looked at how to retrieve small pieces of metadata using the package.xml. But how do we get everything we need in version control in order to use CI/CD for it?

‍

There are a few ways to go about this, including what we looked at earlier, where we have a package.xml that describes what we want to retrieve. Then, we can use the Salesforce DX CLI to issue a retrieve command based on the contents of the package. But what about other metadata? Do you need to modify the package.xml and add every single thing to it, such as flows, email templates, etc.? The answer is no—so let’s look at how you can get that metadata.

The Org Browser

The first way of retrieving metadata that we’ll look at is the org browser, which you can access in the left side menu. This will scan everything in the org that your project is connected to, allowing you to browse the org and download individual pieces of metadata.

‍

‍

For example, let’s say we want to retrieve the “account service layer”, as shown in the image below. We’ll see two options: one to retrieve, and one to retrieve and view. For our example, we’ll retrieve and view it.

‍

‍

This will open up two files: the actual XML file for the Apex class and the real Apex code. If we navigate back to our directory, we’ll also see that this class has now been added.

‍

‍

Behind the scenes, we can assume that the org browser is using the metadata API to retrieve this information and put it in our project.

‍

If we want to do this for all the Apex classes, we can go to the folder and click “Overwrite” on the box that pops up. This will retrieve every single Apex class in our Salesforce org.

‍

‍

Notice that we didn’t have to modify the package.xml at all. In fact, it doesn’t matter if the package.xml is ‘out of sync’ with what is actually in the SFDX project and doesn’t reflect what you want to retrieve.

Populating the Package.XML

Another way we can retrieve metadata is by populating the package.xml. For this one, we’ll create a brand new SFDX project by opening up the command palette in Visual Studio Code and choosing the “Create Project with Manifest” option. “Manifest” refers to the package.xml file we saw earlier.

‍

‍

Once you click on that, choose “standard”, give the project a name (we’ll go with “sample-manifest”), and click “Create Project”.

‍

‍

Like any other SFDX project, this project starts out completely empty save for the base structure. One difference, however, is that there is a “manifest” folder, which contains a package.xml. This package.xml comes with the default configuration for all code-based metadata, such as Apex classes, Visualforce pages, Apex triggers, etc.

‍

Once you authenticate your project to an org, you’ve got a few options. If you right click the package.xml file, there will be an option to “Retrieve Source in Manifest from Org”. This option is only available if we use this specific project template and are connected to an org. It will not be available if it’s connected to a scratch org, so we’ll always make sure we’re using a dev org or playground instead.

‍

‍

Once we’ve done that, we’ll see we have retrieved a lot of code-based metadata, including all the Apex classes and any other code-based metadata that was included.

‍

This method, along with this particular template, is recommended as a starting point. If you’ve never used CI/CD before, it could be overwhelming to have every single thing in version control, so starting with all of the code—triggers, classes, Lightning Web Components, etc.—is a very good start.

The Package.xml Generator Extension

‍

But what if we want to include additional things here, like certain custom fields or certain email templates? Do we need to manually modify this package.xml? Well, yes and no. As we saw earlier, we could use the org browser, which works independently of what’s in the package.xml. But, if we want to have our metadata specifically detailed in the package.xml, there is a Visual Studio Code extension called SFDX Package.xml Generator, that can help us modify the file.

‍

‍

Click on that, and the extension scans your org and pulls all the metadata, allowing you to specify exactly what you want in that package.xml. For example, let’s say we want to include all our custom fields. We’ll search for custom fields, and then, if we don’t want to include all of them, we can choose which ones we want (e.g. only account custom fields).

We then have the option to copy to clipboard or update the package.xml. 

‍

If we choose the latter and then go back to the same package.xml under the Manifest folder, we’ll see that it’s been automatically updated to include our selection. Once we’re here we can right click on either the package.xml or the file itself and again click “Retrieve Source from Manifest in Org”. This will retrieve whatever is in the Manifest into our SFDX project.

The Manifest Create From Org Command

A third (although less recommended) option you can use to retrieve almost everything in your Salesforce org is to use the Manifest Create From Org command. If you’re just getting started with CI/CD, this option is less recommended as it a) creates more complexity when you have everything in version control and b) occasionally retrieves things that cannot be deployed back. However, it’s still an option to have in your toolbelt.

‍

Starting from a project with no metadata, run this command:

‍

‍

This command, Manifest Create From Org, creates a package.xml based on the contents of the org. So, if you want to retrieve everything that’s in the org—fields, classes, customizations—but you don’t know what all there is, you can do that with this command. You’ll have to give the org name or, in our example, the SFDX alias of the Salesforce org, and specify where you want that package.xml to be created. For our purposes, we’ll specify that we want it to be under the Manifest directory, and the file name will be “allMetadata”. Note that it doesn’t need to actually be named package.xml.

‍

After we hit enter, we’ll see that “allMetadata” appears under the Manifest directory and that it has everything that exists in our org.

‍

As mentioned above, this method pulls everything from your org, including objects you may not want in version control or that you didn’t even know existed or are irrelevant. Deploying these objects back to an org can also pose challenges, so be aware of these drawbacks.

Other SFDX Commands

You can also retrieve metadata manually using the SFDX CLI.

‍

Let’s find the force:source:retrieve command in the documentation for the CLI. As you can see, there are multiple variations on this command.

‍

‍

For example, if we wanted to retrieve all the Apex classes in the org, we would use that particular variation of the command, where “m” stands for metadata type.

‍

To give it a try, let’s copy that command, head back to our blank project, and paste and run the command. Just like that, we’ll see all the Apex classes in the org.

‍

We can also retrieve a specific Apex class. This time, we’ll copy the “retrieve a specific Apex class” command from the documentation. This command is the same as the previous one except for the colon and name of the class at the end. If we paste and run that command into our blank project, we’ll retrieve that specific Apex class.

‍

As there are quite a few variations, we recommend going through the documentation to get an idea of what the different commands are. You can also mix and match what you want to retrieve, such as in this example command:

‍

‍

Note: The SFDX commands have now been replaced by SF commands. For example, the sfdx force:source:retrieve command has been replaced by the sf project retrieve start command. That said, the old commands do still work. You can find more information here.

Conclusion

In this guide, we looked at a variety of ways to retrieve Salesforce metadata, including the Metadata API, which is used to retrieve and deploy Salesforce metadata, and the Tooling API, which is used to query specific attributes of a metadata item and build SFDX plugins to extract additional information. We also discussed the various retrieving commands you can use through the SFDX CLI. 

Mastering the essential skill of retrieving Salesforce metadata will enhance your implementation of CI/CD, as well as empower you to more efficiently manage, deploy, and scale Salesforce configurations.