Using 'swift package fetch' in an Xcode project

Up until now, the Cocoa with Love git repositories have included their dependencies as “git subtrees” where each dependency is copied and statically hosted within the depender. I want to replace this arrangement with a more dynamic dependency management while remaining totally transparent to users of the library.

I’d like to use the Swift Package Manager for the task but it’s complicated by the fact that I don’t want the Swift Package Manager to be a required way to build any of these repositories. The Swift Package Manager has a very narrow range of build capabilities and I don’t want my libraries to be subject to these limitations.

In this article, I’ll look at a hybrid approach where the Swift Package Manager is used as a behind-the-scenes tool to fetch dependencies for an otherwise manually configured Xcode project that continues to support the same target platforms and build structures from the previous “subtree” arrangement.

Managing dependencies

The dependency graph for the CwlSignal library that I released last year is pretty simple:

CwlCatchException ← CwlPreconditionTesting ← CwlUtils ← CwlSignal

Git subtree

Until now, I’ve been using Git subtrees.

In this arrangement, you don’t need to separately download any dependencies; if you browse the previous structure of the CwlSignal repository, you’ll notice that it contains CwlUtils, which contains CwlPreconditionTesting, which contains CwlCatchException. This is not totally different to manually copying the files into each repository but with the minor advantage that if a dependency changes, I can easily pull changes into the depender with a simple subtree pull.

This arrangement has its problems. Subtrees scale poorly since changes need to be manually pulled into each subsequent link in the chain; you can’t easily update all dependencies with a single command. Each repository is bloated by needing to contain its dependencies. It also creates merge problems if you accidentally modify a dependency’s subtree in the depender then try to pull the dependency.

None of these are major problems for a small, simple dependency graph like CwlSignal but they’ve always been concerns that I’ve wanted to address.

Git submodules

I could use git submodule. In theory, it’s just a more dynamic approach for the problem that git subtrees solve. I feel as though git submodules should be ideal choice but in practice, git modules are not a transparent change to your git repository and their poor handling by git makes them confusing and frustrating for users.

It is possible to pull and push repositories in the wrong order and end up overwriting your changes. Switching from one dependency to another is complicated and often involves manually editing the contents of the “.git” directory. The “Download ZIP” functionality on Github becomes completely useless as the ZIP file omits the submodule references as do most other non-git means of managing your code.

In comparison to git subtrees which usually go unnoticed, submodules suffer from the fact that every user needs to be aware of the submodule arrangement and needs to run slightly different git commands just to fetch and update the repository correctly.

Established package managers

I could move to an established package manager like CocoaPods or Carthage. While I should probably do more to improve compatibility with these systems for users who want to use them, I’d rather not force everyone to use them. For my own purposes, I’d rather avoid the loss of control over the workspace or build settings imposed by the use of these systems so I’d rather keep a workflow that can be used independent of these systems.

Swift Package Manager

Which brings me to the Swift Package Manager; a combined build-system and dependency manager.

Does the Swift Package Manager offer anything new compared to the previous options I’ve mentioned? Well, it offers a new build system but my primary motivation here is dependency management; I wasn’t looking for a build system.

Using the Swift Package Manager isn’t going to have the weird effects on the repository that git submodules have. It is also bundled with Swift, so it’s a lower friction option than CocoaPods or Carthage – although I still don’t want to force users of my library to use the Swift Package Manager.

Is it possible to use the Swift Package Manager as a dependency resolver without using it as a build system?

Hoping for the future

When I said “my primary motivation here is dependency management; I wasn’t looking for a build system”, I wasn’t being totally honest. I should be primarily motivated by dependency management but truthfully, I’d also like to play around with the Swift Package Manager build system.

Like Apache Maven or Rust Cargo, the Swift Package Manager includes a convention-based build system. While some metadata is declared in a top-level manifest, the build is, as much as possible, determined by the organization of files in the repository. I’m a big fan of this type of build system; a build shouldn’t require substantial configuring. Assuming folder structure conventions are followed, it should be possible to infer most – even all – parameters for a build, rather than forcing the programmer to manually enumerate all aspects, every time.

I would like to see Swift Package Manager projects become a project type in Xcode. Automatically keeping files sensibly in module folders rather than arbitrarily distributed around the filesystem and needing constant sheparding. Build settings inferred rather than configured through a vast array of tabs and inspectors. Dependencies viewable like the Xcode “Debug Memory Graph” display.

Obviously, though, Xcode doesn’t support the Swift Package Manager, yet (Swift Package Manager supports Xcode but it’s the other direction that’s more interesting to me). And, bluntly, until the Swift Package Manager can build apps, build for iOS, build for watchOS, build for tvOS, build mixed language modules, build bundles with resources, manage separate dependencies for tests or inline across modules, it won’t satisfy all the requirements of even a relatively simple library like CwlSignal.

But I’d still like to support the Swift Package Manager as a secondary build option in the hope that it in a couple years it will be possible to make it the primary option.

Swift package fetch

Wanting to use the Swift Package Manager for dependency management but not as the primary build system creates some problems.

Summarizing what need to be done:

  1. Support the Swift Package Manager (for both building and fetching dependencies)
  2. Make Xcode projects refer to the files downloaded by Swift Package Manager.
  3. Make everything transparent to the user.

1. Support the Swift Package Manager

Setting up the “Package.swift” file, adding semantic version tags to repositories and making sure that dependencies are fetched is trivial.

The signficant work was actually across the following tasks (in no particular order):

  1. Reorganize my folders to follow the convention-based structure expected by the Swift Package Manager (all projects).
  2. Separate my mixed Objective-C/Swift modules into separate modules (all projects except CwlSignal).
  3. Under a #if SWIFT_PACKAGE guard, be certain to import the new modules created by the separation in step 2 (all projects except CwlSignal).
  4. Separate my “.h” files so that they can be included from a project-wide umbrella header as in Xcode or from a module header as in Swift-PM (all projects except CwlSignal).
  5. Ensure that symbols affected by step 2 that were previously internal were public so they remained accessible (CwlCatchException).
  6. Remove any reliance on DEBUG or other conditions not set by the Swift Package Manager (CwlDeferredWork and tests in CwlUtils and CwlSignal).
  7. In Objective-C files that needed to both reference and be referenced by Swift, changed the references from Objective-C to Swift to dynamic lookups to avoid circular module dependencies (CwlMachBadInstructionHandler).
  8. Move Info.plist files around. These are generated automatically by the Swift-PM but must manually exist for Xcode – Swift-PM must be set to ignore them all (all projects).

There was also a non-zero amount of effort involved in accepting how the conventions of the Swift Package Manager work and letting it guide some aspects of the build.

2. Make Xcode projects refer to the files downloaded by Swift Package Manager

In Swift 3.0, dependencies are placed in “./Packages/ModuleName-X.Y.Z” where X.Y.Z is the semantic version tag of checkout. Obviously, this will change if you ever change the version depended upon.

Swift 3.1 and later place dependencies in “./.build/checkout/ModuleName-XXXXXXXX” where XXXXXXXX is a hash derived from the repository URL. The hash is not guaranteed to be stable and as far as I can tell, the path format is undocumented and subject to change.

Clearly, we can’t point Xcode directly at either of these since they’re subject to change. We need to create symlinks from a stable location to these subject-to-change locations. This means that we need a simple way to determine the current locations.

The closest we get to a documented solution for handling these paths is the output to the following command:

swift package show-dependencies --format json

The output from this command offers a JSON structure that includes the module names and the checkout paths. While there’s no guarantee that this structure will remain stable in the future, it is currently stable across 3.0 and 3.1 so it’s better than simply enumerating directories.

We need to create symlinks from a stable location to the locations detailed in this JSON file.

I chose to use the location “./.build/cwl_symlinks/ModuleName” to store a symlink to the actual location used by either Swift 3.0 or Swift 3.1 package managers. This creates the minor risk of collision in the scenario where two dependencies have the same module name but different origin or version but outside of that possibility it should offer a stable location that my Xcode projects can use to find these dependencies. When I change the version of a library that I depend upon or the hash changes (because I’m switching between local and remote versions of a repository for testing) or something else that affects the path, all we’ll need to do is update the symlink.

Getting Xcode to keep a path through the symlink rather than immediately resolve the symlink is a little tricky. While using Swift 3.1, I actually created a duplicate of each “./.build/checkout/ModuleName-XXXXXXXX” folder at the “./.build/cwl_symlinks/ModuleName” location, added all the files to Xcode before deleting the duplicate and creating a symlink at “./.build/cwl_symlinks/ModuleName” pointing to “./.build/checkout/ModuleName-XXXXXXXX”.

3. Making everything transparent to the user

We now have two non-transparent steps that we need to eliminate:

  1. Run swift package fetch to get the dependencies.
  2. Create symlinks in a stable location for all dynamically fetched dependencies.

For this, we’ll need some kind of automated script.

Some kind of automated script

We need a “Run script” build phase at the top of any Xcode target with external dependencies. If you have multiple targets with external dependencies, it might be best to do this in its own “Aggregate” target (e.g. named “FetchDependencies”) and have other targets depend on the “Aggregate”.

When you add a “Run script” build phase to Xcode, it defaults to “/bin/sh”. I got about 5 lines into that before remembering that I’m terrible at bash so I change the “Shell” value for the build phase to /usr/bin/xcrun --sdk macosx swift -target x86_64-apple-macosx10.11 (since I need to ensure the macOS SDK is used, even when building targets for iOS or other platforms) and set the script to the following code. Some of the parsing and configuring is a little dense but you should be able to read the comments to get a general feel for what’s happening.

import Foundation

let buildName = ".build"
let symlinksName = "cwl_symlinks"

guard let srcRoot = ProcessInfo.processInfo.environment["SRCROOT"].map({
   URL(fileURLWithPath: $0) }) else {
   print("Environment variable SRCROOT must be set")
   exit(1)
}

/// Launch a process and run to completion, returning the standard out on success.
func launch(_ command: String, _ arguments: [String], directory: URL? = nil) -> String? {
   let proc = Process()
   proc.launchPath = command
   proc.arguments = arguments
   _ = directory.map { proc.currentDirectoryPath = $0.path }
   let pipe = Pipe()
   proc.standardOutput = pipe
   proc.launch()
   let result = String(data: pipe.fileHandleForReading.readDataToEndOfFile(), encoding:
      .utf8) ?? ""
   proc.waitUntilExit()
   return proc.terminationStatus != 0 ? nil : result
}

// STEP 1: use `swift package fetch` to get all dependencies
print("Starting package fetch...")
if let fetchResult = launch("/usr/bin/swift", ["package", "fetch"], directory: srcRoot) {
   if fetchResult == "" {
      print("All dependencies up-to-date.")
   } else {
      print(fetchResult, terminator: "")
   }
} else {
   print("### swift package fetch failed")
   exit(1)
}

// Create a symlink only if it is not already present and pointing to the destination
let symlinksURL = srcRoot.appendingPathComponent(buildName).appendingPathComponent(
   symlinksName)
func createSymlink(srcRoot: URL, name: String, destination: String) throws {
   let location = symlinksURL.appendingPathComponent(name)
   let link = URL(fileURLWithPath: "../../\(destination)", relativeTo:location)
   let current = try? FileManager.default.destinationOfSymbolicLink(atPath: location.path)
   if current == nil || current != link.relativePath {
      _ = try? FileManager.default.removeItem(at: location)
      try FileManager.default.createSymbolicLink(atPath: location.path,
         withDestinationPath: link.relativePath)
      print("Created symbolic link: \(location.path) -> \(link.relativePath)")
   }
}

// Recursively parse the dependency graph JSON, creating symlinks in our own location
func createSymlinks(srcRoot: URL, description: Dictionary<String, Any>, topLevelPath:
   String) throws {
   guard let dependencies = description["dependencies"] as? [Dictionary<String, Any>]
      else { return }
   for dependency in dependencies {
      guard let path = dependency["path"] as? String, let relativePath = (path.range(of:
         topLevelPath)?.upperBound).map({ path.substring(from: $0) }), let name =
         dependency["name"] as? String else {
         throw NSError(domain: "CwlError", code: 0, userInfo:
            [NSLocalizedFailureReasonErrorKey: "Unable to parse dependency structure"])
      }
      try createSymlink(srcRoot: srcRoot, name: name, destination: relativePath)
      try createSymlinks(srcRoot: srcRoot, description: dependency, topLevelPath:
         topLevelPath)
   }
}

// STEP 2: create symlinks from our stable locations to the fetched locations
guard let descriptionString = launch("/usr/bin/swift", ["package", "show-dependencies",
   "--format", "json"], directory: srcRoot) else {
   print("### swift package show-dependencies failed")
   exit(1)
}
do {
   guard let descriptionData = descriptionString.data(using: .utf8), let description =
      try JSONSerialization.jsonObject(with: descriptionData, options: []) as?
      Dictionary<String, Any>, let topLevelPath = (description["path"] as? String).map({
      $0 + "/" }) else {
      throw NSError(domain: "CwlError", code: 0, userInfo:
         [NSLocalizedFailureReasonErrorKey: "Unable to parse dependency structure"])
   }
   try FileManager.default.createDirectory(at: symlinksURL, withIntermediateDirectories:
      true, attributes: nil)
   try createSymlinks(srcRoot: srcRoot, description: description, topLevelPath:
      topLevelPath)
   print("Complete.")
} catch {
   print("### symlink creation failed: \(error)")
   exit(1)
}

Compiling and running this code takes about a second but it would still be better to avoid that overhead after the first run. You can add $(SRCROOT)/Package.swift to the “Input Files” list for the Run Script and add one of $(SRCROOT)/.build/cwl_symlinks/ModuleName (where “ModuleName” is a modules fetched by the task) for each dependency fetched by the Swift Package Manager. This will prevent Xcode re-running unless the “Package.swift” file changes or the module symlink is deleted, eliminating that additional second of overhead.

Irony note: To avoid the static inclusion of dependencies, I’ve statically included this file in each repository.

Try it out

You can inspect or download the CwlCatchException, CwlPreconditionTesting, CwlUtils and CwlSignal projects from github. All dependencies are now fetched using “swift package fetch”.

These projects all now support the Swift Package Manager for building on macOS. Theoretically, the Swift Package Manager opens up the possibility of some of these on Linux but that’ll be an exercise for another day.

This is an experimental change to these repositories. There’s every likelihood that I’ve broken something or ignored a better option, somehow. Create an issue on github if you encounter any problems or have a suggestion for a better approach.

Update: Fetching child projects

I primarily wrote this fetch code to handle CwlSignal’s static inclusion of files from CwlUtils. The key point there is that CwlSignal is not dependent on the CwlUtils project file or any of the CwlUtils targets; the dependency is purely upon some of the sources files within the CwlUtils repository.

Is it possible to handle a dependency on a child project and its targets this way and still keep Xcode happy?

Initially, you might think that a dependency on a child project shouldn’t be any harder than a dependency on a file but it turns out to have a number of complications. To demonstrate, I’ll create an Xcode “Cocoa Application” project that pulls the CwlSignal project as a dependency using “swift package fetch” and builds the CwlUtils_macOS target for linking with the command line tool.

Steps

  1. Create a new “macOS Cocoa Application” project in Xcode, named “TestApp”, language Swift, unit tests enabled.
  2. From Terminal, run “swift package init” in the directory.
  3. Delete the Tests/LinuxMain.swift and Sources/TestApp.swift and move TestApp/ to Sources/TestApp/ (fixing all moved paths in Xcode)
  4. Add dependencies: [.Package(url: "https://github.com/mattgallagher/CwlSignal.git", majorVersion: 1)] to the Package.swift file.
  5. Create a cross-platform “Aggregate” target in the TestApp project named “FetchDependencies” with a run script using shell /usr/bin/xcrun --sdk macosx swift -target x86_64-apple-macosx10.11, the fetch script shown above, an “Input file” of $(SRCROOT)/Package.swift and an “Output file” of $(SRCROOT/.build/cwl_symlinks/CwlSignal
  6. Add a dependency from the TestApp target to the FetchDependencies target.
  7. Build the “FetchDependencies” target
  8. Find where the Swift Package Manager downloaded the dependencies and add the CwlSignal.xcodeproj file to the SignalTest project
  9. Close the project and open the SignalTest.xcodeproj/project.pbxproj file in a text editor.
  10. Replace the location of the CwlSignal project (something like “Packages/CwlSignal-1.1.2/CwlSignal.xcodeproj” in Swift 3 or “.build/repositories/CwlSignal–469046211052243375/CwlSignal.xcodeproj” in Swift 3.1) with “.build/cwl_symlinks/CwlSignal/CwlSignal.xcodeproj”
  11. Save and close “TestApp.xcodeproj/project.pbxproj” and reopen the TestApp project in Xcode.
  12. Add a “Copy Files” phase with Destination “Frameworks” to the TestApp target and add the macOS CwlSignal.framework product from the child CwlSignal project

That’s a lot of steps but it will appear to work – it’s a macOS application that pulls a project dependency using “swift package fetch”.

It’s a little annoying that the FetchDependencies target of the child CwlSignal project gets run in its own directory, fetching all of its dependencies a second time but that’s no the biggest problem.

The problem comes if you delete the dependencies and try to build from this “clean” state. Clean the build folder in Xcode, close the TestApp project in Xcode, delete the “Packages” (if it exists) and run “swift build –clean dist” in the TestApp’s folder to remove the “.build” folder.

Open the TestApp project now and try to build the TestApp. The FetchDependencies target will build successfully but Xcode will report:

Missing dependency target “CwlSignal_macOS (from CwlSignal.xcodeproj)” Swift Compiler error: No such module ‘CwlSignal’

Even though the CwlSignal.xcodeproj was fetched by the FetchDependencies target, Xcode fails to notice and considers it missing.

You need to build a second time for the project to build correctly.

Fetching and building a child project in a single pass

The only approach I’ve found that avoids the need to build twice when fetching .xcodeproj files (and also avoids the multiple dependency fetches) involves removing the CwlSignal.xcodeproj from the TestApp project and instead creating an “External Build System” target in Xcode that builds the CwlSignal_macOS target using xcodebuild. The External Build Tool Configuration will need to be:

* Build Tool: `/usr/bin/xcodebuild`
* Arguments: `-target CwlSignal_macOS -sdk "$(SDKROOT)" $(ACTION) SYMROOT="$(SYMROOT)" OBJROOT="$(OBJROOT)" ARCHS="$(ARCHS)" ONLY_ACTIVE_ARCH=$(ONLY_ACTIVE_ARCH)`
* Directory: `$(SRCROOT)/.build/cwl_symlinks/CwlSignal`

That extended list of “Arguments” is necessary to ensure that dependencies for iOS simulator and device projects are built correctly (otherwise xcodebuild may try to build for macOS or for the wrong device architecture). Obviously, you’d replace “CwlSignal_macOS” and “CwlSignal” with your own dependency’s target name and project name.

With the CwlSignal.xcodeproj no longer part of the TestApp project, there’s no CwlSignal.framework to add to the “Copy Files” phase. To fix this, it is possible to add a CwlSignal.framework file to the project with a location “Relative to Build Products” in the File Inspector. You can add this to the “Copy Files” phase.

This will work correctly, fetching dependencies and building in a single pass. The only major downside is that building a project externally using xcodebuild is slower (there’s a 1 or 2 second overhead every time I hit build with CwlSignal.xcodeproj). It’s not a huge overhead but it’s noticeable versus the practically zero overhead when the CwlSignal.xcodeproj is included as a child project.

Conclusion

I’m glad to remove the git subtree inclusion of dependencies and replace it with something more dynamic.

I’m also happy to have Swift Package Manager support. It’s not Linux support yet (give me time) but it works really smoothly and – other than needing to change a lot of paths – wasn’t particularly difficult.

If it was possible to completely switch to the Swift Package Manager for all use cases, then that would be the end of the story and everything would be a lot cleaner. Unfortunately, current versions of the Swift Package Manager can’t handle a large number of build scenarios (including apps and iOS/watchOS/tvOS platforms) so it’s necessary to keep Xcode as the primary build environment and that means integrating the two.

The “Run Script” build phase works pretty well to hide fetch machinery. Things are going to fail if you don’t have a internet connection when trying to build for the first time, but otherwise it should be transparent and effortless. By setting the “Input Files and “Output Files” for the “Run Script” build step, the minor overhead of compiling and running this step is eliminated in almost all cases so it’s very low interference.

It’s annoying that Xcode can’t detect when a child Xcodeproj file is updated by an earlier target in the build process. Running fetched projects as an external build step isn’t the biggest difficulty ever but it would certainly be nicer if it wasn’t required.

I do have concerns that this build script is liable to break while the Swift Package Manager remains under a high rate of change. I’m sure I’ll need to keep an eye on future Swift Package Manager updates – particularly any that might affect the swift package show-dependencies --format json output.