The Dhall programmable configuration language is now one year old and this post will review progress over the last year and the future direction of the language in 2018.
If you're not familiar with Dhall, you might want to visit the official GitHub project which is the recommended starting point. This post assumes familiarity with the Dhall language.
Also, I want to use this post to advertise a short survey that you can take if you are interested in the language and would like to provide feedback:
Standardization
Standardization is currently the highest priority for the language since Dhall cannot be feasibly be ported to other languages until the standard is complete. In the absence of the standard parallel implementations in other languages are chasing a moving and poorly-defined target.
Fortunately, standardization made good progress this year. The completed parts are:
The one missing piece is the standard semantics for import resolution. Once that is complete then alternative implementations should have a well-defined and reasonably stable foundation to build upon.
Standardizing the language should help stabilize the language and increase the barrier to making changes. The goal is that implementations of Dhall in other languages can review and approve/reject proposed changes to the standard. I've already adopted this approach myself by submitting pull requests to change the standard before changing the Haskell implementation. For example:
This process gives interested parties an official place to propose and review changes to the language.
New integrations
One of the higher priorities for the language is integrations with other languages and configuration file formats in order to promote adoption. The integrations added over the past year were:
- Nix
- JSON and YAML
- Bash
- Text
- i.e. using Dhall as a template engine
Also, an initial Dhall binding to JavaScript was prototyped, but has not yet been polished and published:
- JavaScript (using GHCJS)
- Supports everything except the import system
All of these integrations have one thing in common: they all build on top of the Haskell implementation of the Dhall configuration language. This is due to necessity: the Dhall language is still in the process of being standardized so depending on the Haskell API is the only realistic way to stay up to date with the language.
Dhall implementations in other languages are incomplete and abandoned, most likely due to the absence of a stable and well-defined specification to target:
The JSON integration has been the most successful integration by far. In fact, many users have been using the JSON integration as a backdoor to integrating Dhall into their favorite programming language until a language-native Dhall integration is available.
Some users have proposed that Dhall should emphasize the JSON integration and de-emphasize language-native Dhall bindings. In other words, they believe that Dhall should primarily market itself as an alternative to Jsonnet or HCL.
So far I've struck a middle ground, which is to market language-native bindings (currently only Haskell and Nix) when they exist and recommend going through JSON as a fallback. I don't want to lean for too long on the JSON integration because:
- Going through JSON means certain Dhall features (like sum types and functions) are lost in translation.
- I believe Dhall's future is healthier if the language integrates with a diverse range of configuration file formats and programming languages instead of relying on the JSON integration.
- I want to avoid a "founder effect" problem in the Dhall community where JSON-specific concerns dwarf other concerns
- I would like Dhall to eventually be a widely supported configuration format in its own right instead of just a preprocessor for JSON
New language features
Several new language features were introduced in 2017. All of these features were added based on user feedback:
- Semantic integrity checks
- You can now add an expected SHA-256 hash of an import's normal form to ensure that the behavior never changes
- This is useful for securing Dhall programs that depend on remote imports
- This is also useful for verifying that refactors are behavior-preserving
- String interpolation
- New
constructorskeyword - Currently under review, not officially released- This creates a record containing constructors for all alternatives of a union type
- This improves the ergonomics of using sum types
- Custom headers for URL imports
- This is useful in commercial environments if you need to import Dhall expressions from URLs that require authorization (such as private GitHub repositories)
- Importing expressions from environment variables
- Importing raw
Text- This is useful if you want to import
Textgenerated by other programs (for example: an SSH public key file) without having to preprocess the input to be a valid Dhall string literal (including string escaping)
- This is useful if you want to import
- New
//operator for right-biased non-recursive record merges- This allows users to supply records with default values that can be easily overriden:
{ foo = defaultFoo, bar = defaultBar } // { foo = newFoo }
- New
#operator for concatenating lists - Increase the timeout on remote imports from 1 to 30 seconds
- New
Natural/show,Integer/show,Double/showbuilt-in functions- These are useful for rendering numeric values as
Text
- These are useful for rendering numeric values as
- New
Natural/toIntegerbuilt-in function - New
Optional/buildbuilt-in function- This built-in will fuse with
Optional/fold
- This built-in will fuse with
- Non-empty lists no longer require a type annotation
- Non-empty
mergeexpressions no longer require a type annotation - Union literals no longer require the selected alternative at the beginning
- i.e.
< a : Bool | b = 1 >is now legal syntax
- i.e.
- Imports can now be relative to home directory
- Type annotations now bind more tightly than lambda abstraction
- Disallow reserved punctuation in imported path names: #1 + #2
- This means that paths don't require trailing spaces most of the time
- Identifiers can be escaped with backticks
- This allows identifier names that conflict with reserved keywords
-is now a legal identifier character- Expanded Prelude
- Improvements to error messages (lots of small fixes)
Don't expect as many new language features in 2018. Some evolution is natural in the first year based on user feedback, but I will start rejecting more feature requests unless they have a very high power-to-weight ratio. My goal for 2018 is to slow down language evolution to give alternative implementations a stable target to aim for.
The one likely exception to this rule in 2018 is a widespread user request for "type synonym" support. However, I will probably won't get around to fixing this until after the first complete draft of the language standard.
Tooling
The main tooling features that landed this year were:
dhall-format- Pretty-prints Dhall expressions in a standard format
- this is the Dhall analog of
go fmt - Useful in a commercial environment to avoid stylistic disagreements
- You can also format the
dhallexecutable's output using the--prettyoption
dhall-hash- Computes the semantic hash of any Dhall expression (typically an import)
Tooling is one of the easier things to build for Dhall since command-line tools can easily depend on the Haskell API, which is maintained, well-documented, and up-to-date with the language.
However, excessive dependence on tooling can sometimes indicate a flaw in the language. For example, the recent constructors keyword proposal originated as something that several people were originally implementing using their own tooling and was elevated to a language feature.
Haskell-specific improvements
Dhall also provides a Haskell API that you can build upon, which also landed some notable improvements:
- Marshal some Dhall functions to Haskell functions
- This is useful if you want to customize a Haskell service or program's function logic using dynamically-supplied Dhall code
- This doesn't work for higher-order functions or polymorphic functions
- Derive
Interpretfor Haskell datatypes without field names- Haskell field names without labels correspond to fields named
_1,_2, ... in Dhall
- Haskell field names without labels correspond to fields named
- Expanded set of
Interpretinstances:- Functions (see first bullet point)
- Strict
Text - 2-tuples: #1 + #2
- Linked lists
()
- Customizing conversions between Dhall and Haskell types:
- Dhall API for customizing language
- This lets you define your own variation of Dhall with additional built-in functions
- Add
loadWith - Add
normalizeWith
- Remove dependence on Template Haskell
- This allows cross-compiling the Dhall library
The language core is simple enough that some people have begun using Dhall as a starting point for their implementing their own programming languages. I personally don't have any plans to market or use Dhall for this purpose but I still attempt to support this use case as long as these requests:
- don't require changes to language standard
- don't significantly complicate the implementation.
- don't deteriorate performance
Documentation
The language's official documentation over the last year has been the Haskell tutorial, but I've been slowly working on porting the tutorial to language-agnostic documentation on a GitHub wiki:
The most notable addition to the wiki is a self-contained JSON-centric tutorial for getting started with Dhall since that's been the most popular integration so far.
The wiki will eventually become the recommended starting point for new users after I flesh the documentation out more.
Robustness improvements
I've also made several "invisible" improvements under the hood:
- Fixes to type-checking bugs
- Greatly expanded test suite
- Lots of small bug fixes
- Evaluation performance improvements
- Parsing performance improvements
- Import resolution performance improvements
- Largest improvement - not yet published
I've been slowly whittling away at type-checking bugs both based on user reports and also from reasoning about the type-checking semantics as part of the standardization process.
The next step is to use proof assistants to verify the safety of the type-checking semantics but I doubt I will do that in 2018. The current level of assurance is probably good enough for most Dhall users for now. However, anybody interested in doing formal verification can take a stab at it now that the type-checking and normalization semantics have been formalized.
I also plan to turn the test suite into a standard conformance test that other implementations can use to verify that they implemented the standard correctly.
Infrastructure
This section covers all the miscellaneous improvements to supporting infrastructure:
- Dedicated GitHub project
- Convenient link to latest Prelude: http://prelude.dhall-lang.org/
- Binary releases for Linux and OS X
- NixOps configuration for mirroring Prelude
- This allows commercial users to improve the resilience of the Prelude by hosting their own IPFS mirror
Acknowledgments
I'd also like to thank everybody who helped out over the past year, including:
- bosu
- Ville Tirronen
- Markus Hauck
- Hardy Jones - For contributing binary releases
... and also everybody who filed issues, reported bugs, requested features, and submitted pull requests! These all help improve the language and no contribution is too small.
Conclusion
In 2017 the focus was on responding to user feedback as people started to use Dhall "in anger". In 2018 the initial plan is to focus on standardization, language stabilization, and creating at least one complete alternative implementation native to another programming language.
Also, don't forget to take the language survey if you have time. I will review the feedback from the survey in a separate follow-up post and update the plan for 2018 accordingly.
No comments:
Post a Comment