python|使用Typescript接口进行运行时JSON类型检查 json|接口|java

This blog was originally posted in Dutch on Bytecode’s Insights and in English on my personal website
该博客最初在 Bytecode的见解中 以荷兰语发布， 在我的 个人网站 上以英语 发布。
【python|使用Typescript接口进行运行时JSON类型检查】Bytecode is currently working on a project for a startup. We are developing an app that is linked to a backend data storage service. For this project, I wanted to have more security around the API calls and validate the types at runtime.
字节码目前正在为启动项目工作。我们正在开发与后端数据存储服务链接的应用程序。对于这个项目，我希望在API调用方面具有更高的安全性，并在运行时验证类型。
背景 (Background) At Bytecode, we use Typescript extensively for front-end and mobile development, to prevent Javascript’s liberal dynamic typing system from causing errors. Typescript adds an extra layer of security. In recent years, we have seen a strong decrease in type-related errors due to the use of Typescript.
在Bytecode，我们将Typescript广泛用于前端和移动开发，以防止Javascript的自由动态键入系统引起错误。 Typescript增加了一层额外的安全性。近年来，由于使用Typescript，与类型相关的错误已大大减少。
However, Typescript also has it’s limitations, the main one being that types and interfaces disappear during compile-time. So there are no run-time checks on external data that is not available during compilation, like for example API responses. Ideally, Typescript would support marshalling in a way similar to Go, to secure type safety during runtime. Unfortunately, however, the Typescript layer disappears during compile-time and only Javascript remains: code that knows nothing about the types and interfaces defined in the source code. Unfortunately, the Go-like approach would never be possible using features built into the language.
但是，Typescript也有其局限性，主要的原因是类型和接口在编译时会消失。因此，不会对编译期间不可用的外部数据进行运行时检查，例如API响应。理想情况下，Typescript将以类似于Go的方式支持编组，以确保运行时的类型安全。但是，不幸的是，Typescript层在编译时消失了，只剩下Javascript：对源代码中定义的类型和接口一无所知的代码。不幸的是，使用语言内置的功能永远无法实现类似Go的方法。
Until recently, we did our API calls directly within Redux actions, which didn’t cause any problems for small applications. However, this setup is not the best, considering the single-responsibility principle and keeping in mind that Redux actions can get confusing with large projects, if you’re not careful. This is why we recently switched to creating separate API packages as an abstraction layer on top of the API calls. We now only call a function that executes the API calls and checks HTTP errors. If there are no errors, we’ll get the data back. If there are errors, an error is thrown. This way, the Redux code does not know anything about the details of the API call.
直到最近，我们还是直接在Redux操作中进行了API调用，这对于小型应用程序没有任何问题。但是，考虑到单一责任原则，并且记住Redux操作可能会使大型项目感到困惑，因此这种设置并不是最佳选择。这就是为什么我们最近切换到创建单独的API包作为API调用之上的抽象层的原因。现在，我们仅调用执行API调用并检查HTTP错误的函数。如果没有错误，我们将取回数据。如果有错误，将引发错误。这样，Redux代码对API调用的详细信息一无所知。
Another reason to use a separate API package, is our desire to be able to set up an SDK more easily later on, when we start targeting another platform (think of an application, first only web, but where a mobile app will be added as well). It will then be possible to use shared code easily, without duplicate logic. Since these API packages become increasingly important with more dependent applications, the run-time guarantees also get more crucial. If the SDK says that a function returns a certain data type, we also want to guarantee this or else give an error message.
使用单独的API程序包的另一个原因是，我们希望以后能够更轻松地设置SDK，这是当我们开始定位到另一个平台时(想一个应用程序，首先是Web，但是将移动应用程序添加为好)。这样就可以轻松使用共享代码，而无需重复逻辑。由于这些API软件包在具有更多依赖性的应用程序中变得越来越重要，因此运行时保证也变得越来越重要。如果SDK指出函数返回某种数据类型，我们也要保证这一点，否则会给出错误消息。
需求与研究 (Requirements and research) Our research question consisted of several parts:
我们的研究问题包括以下几个部分：

How can we generically check a JSON object against a Typescript interface, without the need for duplicate code for type definitions?
我们如何在Typescript接口上一般检查JSON对象，而无需为类型定义重复代码？
How can we achieve the above without having to modify other production code outside the API package to make this check possible?
我们如何在不修改API包之外的其他生产代码的情况下实现上述目标呢？
How can this be done in NodeJS, React Native and in the browser?
如何在NodeJS，React Native和浏览器中完成此操作？

There are enough libraries that make it possible to check a JSON structure, based on a DSL (domain specific language). However, this wasn’t what we were looking for, because we were already using Typescript and didn’t want to maintain the same type definition in multiple ways. We would prefer to develop a solution where no code generation or extra step in compilation is needed, but everything on-the-fly during runtime (like Go).
有足够的库可以基于DSL(特定于域的语言)检查JSON结构。但是，这不是我们想要的，因为我们已经在使用Typescript，并且不想以多种方式维护相同的类型定义。我们希望开发一种解决方案，其中不需要代码生成或编译中的额外步骤，而需要在运行时即时进行所有操作(例如Go)。
A few months ago, I read a blog post by Picnic, describing their project “Aegis”, in which they had offered a solution to this problem. However, I noticed that it was still difficult to implement. The code is open source, but there was no example of implementation on a larger scale, because this was done within Picnic’s proprietary app. A code generation step was also needed. We would, if possible, prefer not to have this extra step.
几个月前，我阅读了Picnic的博客文章，描述了他们的项目“ Aegis”，他们在其中提供了解决此问题的方法。但是，我注意到仍然很难实施。该代码是开源的，但是没有大规模的实现示例，因为这是在Picnic的专有应用程序中完成的。还需要代码生成步骤。如果可能的话，我们宁愿不要执行此额外步骤。
On the Subreddit of Typescript I had placed a post, where I submitted my question. I primarily received responses with examples of code generation solutions. Some responses discussed runtime solutions, but these solutions were unnecessarily complex and/or required modifications within the build configuration of Typescript. We prefer to avoid this, because we prefer to keep something experimental separate from the rest of our production code, so that if we are not satisfied, we can revert the changes.
在Typescript的Subreddit上，我张贴了一个帖子，向我提交了问题。我主要收到有关代码生成解决方案示例的回复。一些响应讨论了运行时解决方案，但是这些解决方案在Typescript的构建配置中不必要地复杂和/或需要修改。我们宁愿避免这种情况，因为我们更愿意将实验内容与其余的生产代码区分开来，这样，如果我们不满意，则可以还原所做的更改。
A possible solution that popped into my head was the following:
出现在我脑海中的一个可能的解决方案是：

Load all type-definitions through the file system as strings
通过文件系统作为字符串加载所有类型定义
Use the Typescript compiler as production dependency and parse these strings
使用Typescript编译器作为生产依存关系并解析这些字符串
Compare the result of parsing against the JSON data to see if it matches the interfaces
将解析结果与JSON数据进行比较，以查看其是否与接口匹配

However, this would mean that a substantial part of the Typescript compiler would have to become part of the app and thus increase the bundle size. The Typescript compiler is not the fastest in the world either, so this would take a considerable amount of extra time when it has to be done on-the-fly. In addition, the filesystem is only suitable for Node.js and not for browser environments, so compatibility could not be maintained. Unfortunately this solution was not feasible.
但是，这意味着Typescript编译器的很大一部分将必须成为应用程序的一部分，从而增加捆绑包的大小。 Typescript编译器也不是世界上最快的，因此当需要即时完成时会花费大量的额外时间。另外，该文件系统仅适用于Node.js，不适用于浏览器环境，因此无法保持兼容性。不幸的是，这种解决方案不可行。
Ultimately, I chose to use Picnic’s Aegis, mainly because of the simplicity of the tool and because it can be used without modifying other aspects of the project (compilation steps, configurations or production code in other parts of the application).
最终，我选择使用Picnic的Aegis ，主要是因为该工具的简单性，并且可以在不修改项目其他方面(编译步骤，配置或应用程序其他部分中的生产代码)的情况下使用它。
实作 (Implementation) The final implementation is as follows. Within the API folder, all public types (that is, the arguments and return types of the entire API package) are defined in the types folder. For all types in this folder, Aegis creates decoders and stores them in the internal folder of the API package.
最终的实现如下。在API文件夹中，所有公共类型(即整个API包的参数和返回类型)都在types文件夹中定义。对于此文件夹中的所有类型，Aegis都会创建解码器并将其存储在API包的internal文件夹中。
In order to make the aforementioned solution usable for Bytecode, a few adjustments had to be made in Aegis. For example, we added ESLint comments at the top of the file. This was done in a fork of Aegis on Github. This is the dependency used in Bytecode’s project. To build the decoders, a command has been added to the package.json of the React-Native/Expo project. By simply running yarn run aegis, Aegis is called with the right arguments and all decoders are built.
为了使上述解决方案可用于字节码，必须对Aegis进行一些调整。例如，我们在文件顶部添加了ESLint注释。这是在Github上的神盾之叉中完成的。这是字节码项目中使用的依赖项。为了构建解码器，已将命令添加到React-Native / Expo项目的package.json中。通过简单地运行yarn run aegis ，可以使用正确的参数调用Aegis，并构建所有解码器。
The production code of the API package already used an internal returnOrThrow function, which received an internal API response type (consisting of the response of the API and/or an error if it occurred), threw an error if it existed and otherwise returned the data. This function has been modified so that a second argument is given to the function, namely the decoder. In returnOrThrow the decoder is then used to check the data before it is returned. See the example below:
API包的生产代码已经使用了内部returnOrThrow函数，该函数接收内部API响应类型(由API响应和/或错误(如果发生)组成)，如果存在则抛出错误，否则返回数据。对该函数进行了修改，以便为该函数提供第二个参数，即解码器。在returnOrThrow ，解码器然后用于在返回数据之前检查数据。请参阅以下示例：

import { Decoder } from "decoders/types"; import { guard } from "decoders"; interface APIResultSuccess { data: T; error?: undefined; } interface APIResultFailure { data?: undefined; error: string; } type APIResult = APIResultSuccess | APIResultFailure; const throwOrReturn = (result: APIResult, decoder: Decoder): T => { if (result.error) { throw new Error(result.error); } // We can assume that data is valid (type T) if no error was found const data = https://www.it610.com/article/result.data as T; const decodeChecker = guard(decoder); const _ = decodeChecker(data); // Throws if it's not valid return data; }; export default throwOrReturn;

If the return body does not satisfy the decoder, an error is thrown, which can be caught when calling the API call.
如果返回主体不满足解码器的要求，则会引发错误，在调用API调用时可以捕获该错误。
心愿单 (Wishlist) For now, the implementation of JSON type checks is still experimental. Within the codebase, this type check only affects a small (separate) part, so it can be easily removed later on. This is why it is not automated yet, something we would like to add later on. Automating this without modifying the build configuration can still be a challenge. Until we automate this completely, we can add a check in the CI pipeline that gives an error message when running Aegis causes file changes in Git, meaning the decoders have not been updated after making changes to the type definitions.
目前，JSON类型检查的实现仍处于试验阶段。在代码库中，此类型检查仅影响一小部分(单独的部分)，因此以后可以轻松将其删除。这就是为什么它还没有自动化的原因，我们希望稍后再添加。在不修改构建配置的情况下实现自动化仍然是一个挑战。在完全自动化之前，我们可以在CI管道中添加检查，以在运行Aegis导致Git中的文件更改时给出错误消息，这意味着在更改类型定义后未更新解码器。
Another very good use-case of this setup would be the end-to-end testing of the API for which the package is built. The end-to-end testing of APIs is something that has been on the Bytecode wishlist for some time now. JSON type checking can also provide great added value, allowing you to immediately check whether the API returns data according to the expectations.
此设置的另一个很好的用例是构建该软件包所用的API的端到端测试。 API的端到端测试已经存在于字节码愿望清单上一段时间了。 JSON类型检查还可以提供巨大的附加值，使您可以立即检查API是否根据期望返回数据。
Regarding improvement to Aegis itself, at the moment Aegis is mainly built for the “happy flow”, there are still some edge cases that don’t quite work. The tool is now definitely usable, but improvements are still needed for large-scale use.
关于“宙斯盾”本身的改进，目前“宙斯盾”主要是为“幸福之流”而建造的，但仍有一些边缘案例无法奏效。该工具现在肯定可用，但仍需要大规模使用进行改进。
Support for Aegis configuration files is another useful addition. Now arguments for importPath and outputFile must be given as CLI options. Simply calling aegis generate, which then loads the configuration by itself would be a nice addition.
对Aegis配置文件的支持是另一个有用的补充。现在，必须将importPath和outputFile参数作为CLI选项给出。只需调用aegis generate ，然后再由其自身加载配置，将是一个不错的选择。
If it turns out that this workflow works very well for Bytecode, there is a good chance that we will further develop the Aegis tool ourselves, open-source of course. For now, the tool is not yet available on NPM. That would be the first step towards a stable release.
如果事实证明该工作流程非常适合Bytecode，则很有可能我们将自己进一步开发Aegis工具，这当然是开源的。目前，该工具尚未在NPM上提供。这将是迈向稳定版本的第一步。
示例项目 (Example project) As an addition to this article, an example project is also available, with a simple API call, where the tool can be seen in action.
作为本文的补充，还提供了一个示例项目，该项目带有一个简单的API调用，可以在其中看到该工具的实际作用。
Check out the project on Github.
在Github上检查项目。