Rostyslav Pidburachynskyi

Never sort by yourself

Fri, 05 Apr 2024 00:00:00 +0000

Never sort by yourself

Preamble

Sorting is a staple action in coding. The demand to organize data in either ascending or descending order is frequent, and often, we are faced with the challenge of morphing the same data to match different sorting needs before displaying it to the user.

Sorting becomes more complicated when dealing with massive volumes of data; however, we can’t do this client-side. So, we must resort to performing this action server-side. But is it something we should be handling on our own?

Let’s explore it further.

The problem

Imagine this scenario, you have a basic REST endpoint that return a list of users:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


[
 {
 "id": 1,
 "name": "John",
 "age": 20,
 "email": "john.doe@gmail.com"
 },
 {
 "id": 2,
 "name": "Jane",
 "age": 25,
 "email": "jane.doe@gmail.com"
 },
 {
 "id": 3,
 "name": "Bob",
 "age": 30,
 "email": "bob.doe@gmail.com"
 }
]

You’ve also got client software that utilizes this endpoint to show the user list. The issue? This list has to be sorted by the name field in ascending order. So you’ve got two choices:

Sort data on the client side.
Sort data on the server side.

The first approach is relatively straightforward, but it carries a significant disadvantage:

Sorting on the client side can be extremely slow or even impossible when dealing with large datasets.

The second option may appear ideal, but carries its own shortcomings:

Additional endpoints are necessary for every unique client application view.
New endpoints will be needed for every field/order sorting requirement.

The solution

The solution revolves around the utilization of a query parameter that allows us to indicate the preferred sorting field and order. Let’s denote this as sort. The request might then appear like:

1

GET /users?sort=name:asc

Or if you need to sort by age field in descending order:

1

GET /users?sort=age:desc

By adopting this approach, you can sort data server-side and deliver sorted data to the client without the need for creating new endpoints for every field/order sort.

What about multiple-field sorting?

For multi-field sorting, you may consider the following format:

1

GET /users?sort=name:asc,age:desc

Alternate Query Parameter Representations

There exist differing yet commonly adopted methods to represent sorting in query parameters:

sort=name:asc,age:desc - This is undoubtedly the most comprehensible and highly adopted method, offering flexibility in adding more order directions (like asc-null-first, desc-null-last, etc.). You can choose any delimiter in place of :.
sort=name,-age - Although this method is also well-liked, it offers less flexibility in adding more order directions. Nonetheless, it’s compact and quite readable, with the - sign, optionally replaceable with < or > signs. The + sign is always redundant.

Conclusion

Sorting data is a staple action in coding, but it can pose challenges when dealing with large volumes of data, especially when different client-side applications require different sorting needs.

Sorting data on the client-side can be slow or even impossible with large datasets, and server-side sorting often necessitates the creation of additional endpoints for unique client-side views.

However, the use of a query parameter for indicating the preferred sorting field and order can offer a solution. By using this approach, data can be sorted server-side and delivered sorted to the client without the need to create new endpoints for every field/order sort. This provides a flexible, efficient, and scalable solution for sorting data in coding.

The Importance of Unique Sorting in Databases

Wed, 20 Dec 2023 00:00:00 +0000

The Importance of Unique Sorting in Databases

Introduction

In the fast-evolving world of information technology, data has become the lifeblood of businesses. Databases have emerged as a critical component in the information architecture of any organization, enabling optimization and effective management of this data. Two prominent types of databases that have found widespread use are SQL and NoSQL databases.

While SQL and NoSQL have proved to be powerful tools, they are not without their challenges. A common and potentially troublesome issue that many developers have faced is the inconsistent ordering of records despite using the same offset and limit. This problem can be particularly vexing when sorting records by a factor that includes duplicates, such as names - where people with the same names can appear in a different order in different instances.

This discrepancy may seem minor, but it can have wide-ranging implications for the predictability and reliability of any data-driven operations, leading to potential complications in multiple processes such as data analysis, record keeping, and the most import - user experience.

In the ensuing sections of this article, we will delve into this issue in more detail, understanding its causes and exploring potential solutions to ensure consistency in your databases and boost the overall efficacy of your data management.

Concept of Sorting, Offset, and Limit

Sorting, offset, and limit are crucial concepts in database management that allow developers to organize and retrieve data effectively.

Sorting is the arrangement of data in a particular order based on certain criteria. In SQL and NoSQL databases, you can sort data in ascending or descending order based on columns like names, IDs, or any other attribute. For example, if you sort by name, the database will arrange records alphabetically.

Offset and limit, on the other hand, are fundamental components for handling database pagination. The limit dictates how many records we aim to retrieve from the database, while the offset instructs the database where to start the retrieval. For instance, an offset of 10 and a limit of 5 means start from the 11th record and fetch five records.

Example of a SQL query that incorporates sorting, offset, and limit:

1
2
3
4


SELECT * FROM employees
ORDER BY lastname ASC -- Sorting in ascending order
LIMIT 5 -- Limiting the result to 5 records
OFFSET 10; -- Starting from the 11th record

Both offset and limit come in handy when working with large sets of data, as returning everything at once can be inefficient and resource-taxing. However, issues can arise where the same offset and limit do not always result in fetching records in the same order which can lead to data inconsistency. The subsequent sections will delve deeper into this problem.

Different Data Despite Identical Offset, Limit and Sort Criteria

That’s issue this article is aimed at.

A perplexing anomaly that developers often encounter when dealing with databases involves retrieving different sets of data even when using the same offset, limit, and sort criteria.

This inconsistency primarily arises when sorting involves a column with duplicate values, such as names. For instance, if you require records sorted by name, and your database contains multiple entries with the same name, repeated fetch requests may yield variations in the order of returned results. SQL and NoSQL databases don’t guarantee a fixed sequence of items with identical sorting values, leading to unexpected variations in the record sequence. More to say, the database will return them in the order which is the most efficient for the database.

Let’s consider an example. Suppose you have a table of employees with the following records:

ID	Name	Age
1	John	20
2	Jane	25
3	Bob	30
4	John	35
5	Jane	40
6	Bob	45

If query this table with the following SQL query:

1
2
3
4


SELECT * FROM employees
ORDER BY name ASC
LIMIT 3
OFFSET 0;

We will get the following result:

ID	Name	Age
3	Bob	30
6	Bob	45
2	Jane	25

ID	Name	Age
3	Bob	30
6	Bob	45
5	Jane	40

Or even more, running the same query multiple times can yield get each time either of the above results.

Such inconsistency can become problematic, especially where the sequence of data is important, such as in pagination, data comparison, or maintaining data synchronization in real-time applications.

Consider a scenario in which you have a table spread across several pages. You might notice that the same records appear on both the 2nd and 3rd pages. This repetition can push needed records out of view, causing them to go missing on these pages. So, instead of seeing new records as you turn the pages, you’re presented with duplicated entries, and some expected records are nowhere to be found.

The Impact of this Problem

The consequence of retrieving fluctuating orders of records, despite having the same offset, limit, and sort criteria, can significantly affect the reliability and predictability of data-related operations. Below are a few specific areas where this inconsistency can cause complications:

Data Analysis: For data scientists and analysts, consistency is key. Inconsistent order of delivery of data can lead to distorted analysis results, impacting their capability to make informed decisions.
User Experience: In customer-facing applications, this inconsistency can lead to poor user experience. For example, in an e-commerce platform using pagination, customers might see products in a different order each time they visit the same page.
Data Synchronization: In real-time applications, it’s crucial to maintain synchronized data across different platforms. Inconsistent ordering can disrupt this synchronization, leading to discrepancies.
Record Keeping: Consistency is also vital in maintaining organized and reliable records, especially for activities such as audit trails.

Addressing this issue of inconsistency, therefore, is not only about improving database operations but impacts the full system. The following section will explore potential solutions to this problem and ways to ensure database consistency.

Solutions to the Problem

While there are numerous potential solutions to deal with data inconsistency, such as implementing a standardized algorithm or manipulating indexes, they may not always be feasible. The roots of this issue lie in the database structure and its inherent features, which often can’t be changed without complexity. Therefore, a reliable and straightforward way to address this problem is presented here.

The recommended solution is to always incorporate a secondary sorting criterion aimed at enhancing the uniqueness of the sorting process. For example, when sorting by a common field like names, which tend not to be unique, it’s beneficial to add an additional field. This additional field could vary from second names, year of birth, or even their department, making the resultant order increasingly unique.

Nevertheless, the quest for unique sorting can be challenging, and there could still be instances of inconsistency. In such cases, adding sorting at the end by a universally unique field, which is always present in SQL and NoSQL databases - the ID, can help. IDs are always unique, hence sorting by IDs will ensure a consistent order, regardless of how many times you run the query.

Example of a SQL query that incorporates secondary sorting criteria:

1
2
3
4


SELECT * FROM employees
ORDER BY name ASC, age DESC, employee_id ASC
LIMIT 5
OFFSET 10;

In the above example, we have three sorting criteria - name, age, and employee_id. The first two criteria potentially can have duplicate values, while the third one is always unique. This ensures that the sorting process is increasingly unique, and the resultant order is consistent.

Furthermore, you can add timestamp fields, such as date created or date updated, to the sorting criteria, aligning it more with user experience. However, it’s important to note that this might still leave a small room for inconsistency. If your records encompass a compound unique value, sorting via this compound value could be the most foolproof way to ensure database consistency.

Performance Impact

When mitigating the inconsistency problem with additional sorting criteria, a common concern might be the potential impact on database performance. It’s important to clarify that these additional measures for ensuring a unique sort order offer a balance between consistency and operational efficiency.

SQL and NoSQL databases are generally equipped to handle multiple sorting criteria efficiently. They use indexing strategies to optimize the sorting of records, ensuring that any performance overhead caused by introducing additional fields into the sorting mix is usually minimal.

When considering database performance, the impact of additional sorting criteria becomes relatively inconsequential when dealing with low records of duplicate values. If the majority of your data is unique or if you’re operating with minimal duplicate values, the database does not have an extensive sorting task to carry out.

Given this circumstance, the process of sorting primarily runs through unique records, requiring less computational effort. As a result, the database performance remains almost unaffected, ensuring smooth and efficient operations alongside better data consistency and reliability. Thus, even with additional sorting criteria, databases with fewer duplicates continue to maintain optimal performance while ensuring constant order in record retrieval.

Conclusion

In this article, we delved into a common, yet perplexing issue that developers often encounter - the inconsistency of record order despite maintaining the same offset, limit, and sort criteria. This challenge, mainly when dealing with duplicate values during the sorting process, can cause hiccups in database and application reliability.

A proposed solution to this problem emphasizes enhancing the ‘uniqueness’ of the sorting operation. Incorporating secondary sorting criteria or utilizing unique identifiers and compound fields are recommended strategies to constantly ensure a predictable and consistent sorting order.

While it is important to consider potential performance implications, the effect on modern database systems is typically marginal, especially when dealing with low-duplicate datasets.

In conclusion, to reinforce data consistency and avoid potential bugs, it’s strongly recommended to implement these additional sorting measures as a standard practice in your database operations. It not only eliminates uncertainties but adds robustness to your data management endeavors.

The Art of Assertive Programming

Sun, 10 Dec 2023 00:00:00 +0000

Cannot read property of null or The Art of Assertive Programming

Introduction

Imagine yourself orchestrating a large family meal. Numerous dishes are simmering on the stove and roasting in the oven, each requiring your undivided attention. It’s not just about stirring the pot or monitoring the oven timer, it’s about making sure the soup has the right hint of salt or that the roast isn’t charring into a burnt mess. Your vigilance is needed - you can’t afford to wait until each dish is cooked to confirm it’s done right. Real-time checks are vital, ensuring any errors are caught immediately, not after the fact. In essence, this culinary roller-coaster demands nothing less than your assertive attention.

The problem

In our journey as programmers, it’s quite common to work with variables that we trust to always be appropriately defined - we assume they come equipped with a preset value, a specific type, a well-structured form, or even particular restrictions, such as being either positive or negative. But what if our expectations aren’t met?

What happens if, instead of receiving an expected value, we get null? What if a string shows up where we expected a number? What becomes of our computations when we encounter zero in our division operations? Or what if we attempt to access array properties from a string, especially when that string also happens to have a property of the same name? Suddenly, we grapple with what we term undefined behavior, an unpredictable state of our program which seems to diverge all established norms.

Sure, employing a typed language can help alleviate some of these issues, particularly where incorrect types are in play. But how about the other challenges? Realistically, we can’t completely ward off such occurrences since the future remains unpredictable. However, this article isn’t focused on preventing these scenarios, but rather, seeks to address the ideal way to manage them when they arise.

You might be led to think occasional errors are acceptable, given you can always debug the program later. However, what if you had tools to simplify your debugging process? Consider a typical error like Cannot read property 'name' of null. The origin of this error is usually easy to pinpoint through the stack trace, as with most errors of a similar nature. But what about a scenario where the function completes without throwing an error, albeit yielding an incorrect result? Identifying the source of such a problem could prove more difficult and require more time to debug.

Let’s take a look at some cases to illustrate the problem, to help understand across many languages, the cases are expressed as pseudo-code.

Case 1

Suppose we have a function that accepts a number and returns its power of two.

1
2


function square_root x
 return x * x

If we were to call this function with a string, we would get an error.

Case 2

Now, let’s consider another example. Suppose we have a function that accepts two numbers and returns their sum.

1
2


function add x, y
 return x + y

If we were to call this function with a string and/or number, we would get a successful result as most languages will coerce the number to a string and concatenate the two values.

Case 3

Now, let’s consider another example. Suppose we have a function that accepts a number and returns its square root.

In reality square root of a negative number exists.

1
2


function square_root x
 return sqrt(x)

If we were to call this function with a negative number, we would get an error.

The solution

The solution to this predicament lies in actively monitoring and validating your variables and input arguments. When a function receives an input or an argument, you can establish checks to confirm that it meets your criteria. Should it fall short, your program could respond by immediately throwing an error.

This technique is often referred to as an assertion. An assertion serves as a powerful instrument to bolster the robustness, security, and efficiency of your code. This systematic scrutiny of variables and arguments allows for proactive error detection, making your code not only more resistant to unpredictable behaviour but also simpler to debug and maintain.

What is assertion?

An assertion is a statement that evaluates to either true or false. It’s a way of expressing a condition that must be true at a particular point in your program. If the condition is false, an error is thrown.

The pseudo-code below illustrates the concept of an assertion.

Examples

1

assert <condition> <message?>

The condition is the expression that must evaluate to true. If it evaluates to false, the message is thrown as an error. The message is optional, but it’s a good practice to include it, as it provides additional context to the error.

Let’s look at a simple example:

1
2


a = random(1, 10)
assert a > 5, "a must be greater than 5"

In this example, we generate a random number between 1 and 10 and assign it to the variable a. We then assert that a must be greater than 5. If a is less than or equal to 5, an error is thrown with the message “a must be greater than 5”. This example does not make sense in real program, but it illustrates the concept of an assertion.

Here is a better example:

1
2
3


function divide a, b
 assert b != 0, "b must not be zero"
 return a / b

In this example, we assert that b must not be zero. If b is zero, an error is thrown with the message “b must not be zero”. This assertion prevents the function from dividing by zero, which would otherwise result in another error unhandled error.

And the last example:

1
2
3
4


function add a, b
 assert typeof a == "number", "a must be a number"
 assert typeof b == "number", "b must be a number"
 return a + b

In this example, we assert that a and b must be numbers. If either a or b is not a number, an error is thrown with the message “a must be a number” or “b must be a number”. This assertion prevents the function from adding non-numbers, which would otherwise execute successfully but yield an incorrect result.

When to use assertion?

Assertions are best used to validate input arguments and variables that come from an external source which are used for internal purposes. For example, the following is a correct use of assertion:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


function check_user_verified_email user_id
 # user_id comes from an arguments
 assert typeof user_id == "string", "user_id must be a string"

 # call to a database is an external source
 user = db.find_one("users", { email: email })

 assert user.email != null, "user email must not be null"
 assert user.email != "", "user must have an email"

 return user.email.starts_with("verified")

In this example, we assert that user_id is actually a non-empty string. We then assert that the user must have an email because the function is not supposed to be called for user without email. These assertions prevent the function from executing with invalid input arguments or variables.

Note: We check user.email only because the function expected to be called only for users with email, if it was expected to be called for users without email, we should not check it.

Note: Also we can check if we suppose that email should be present in user object.

Why did we use assertion for checking user email? Read more about it in the next section.

Should I use assertion for external data grabbing?

Short answer - yes, long answer - it depends. Here is a real example you can better understand it with.

Consider a scenario where you possess an orders table in your database comprising standard fields, such as order_id, customer_id, order_date, product_id, and quantity. You are in need to introduce a new field named status, an enum accommodating the values pending, shipped, or delivered. You would set this field, by default, as pending for all new orders. A certain department has assured you that they would assign the status to the existing orders. Meanwhile, there’s a function that currently runs fine, calling upon each order daily to perform specific operations. Now, your task is to adjust this code to make use of the status for some corresponding logic. Here is that method:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


function process_order order_id
 order = db.find_one("orders", { order_id: order_id })

 # do some logic
 ...

 # new added logic
 if order.status != "delivered"
 # order.status now can be 'pending', 'shipped' or null
 send_email_to_customer_about_order_status(order_id, order.status)

Essentially, the existing code aligns with the older logic and is likely to function smoothly since it doesn’t utilize the new status field. Post incorporating the new logic, we aim to email customers about all order statuses with the exception of the delivered ones. However, we could potentially face an issue if the status field isn’t defined. Even though assurances have been received stating this wouldn’t happen, we must prepare for the possibility.

Directly sending a null value to the function isn’t feasible, as it would result in undefined behaviour. Similarly, neglecting it ain’t an option as it would leave the order unprocessed. In this complex situation, the most suitable alternative is to throw an exception and inspect that particular order to understand why it lacks a status.

Indeed, we could insert an if statement to perform the respective action. Yet, if we envision a function with numerous if statements, it could make the code convoluted and challenging to maintain.

In contrast, using assertions can effectively condense each statement to a single code line, as shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13



function process_order order_id
 order = db.find_one("orders", { order_id: order_id })

 # do some logic
 ...

 # it checks if the status is not null
 # else it throws an exception with appropriate message
 assert order.status != null, "order status must not be null"

 if order.status != "delivered"
 send_email_to_customer_about_order_status(order_id, order.status)

When not to use assertion?

For control flow

Assertions are not meant to be used for control flow. For example, the following is an incorrect use of assertion:

1
2
3
4
5
6


function add a, b
 assert typeof a == "number", "a must be a number"
 assert typeof b == "number", "b must be a number"
 if a < 0
 assert b < 0, "b must be negative"
 return a + b

In this example, we assert that a and b must be numbers. We then assert that b must be negative if a is negative. This is an incorrect use of assertion because it’s being used for control flow. Instead, we should use a conditional statement:

1
2
3
4
5
6
7


function add a, b
 assert typeof a == "number", "a must be a number"
 assert typeof b == "number", "b must be a number"
 if a < 0
 if b >= 0
 throw "b must be negative"
 return a + b

In this example, we assert that a and b must be numbers. We then check if a is negative. If a is negative, we check if b is negative. If b is not negative, we throw an error. This is a correct use of assertion because it’s not being used for control flow.

The example provides an explanation of what control flow is within the context of a method, but what are the actual disadvantages of employing assertions for managing control flow?

Drawbacks of using assertions for control flow

Assertions can be disabled globally - it’s very popular behaviour, so you can’t rely on them with real data.
Poor handling of errors - usually you can’t handle assertion errors properly so you just log them to debug later, and you won’t be able to notify client about the error.
Bad user experience - you can’t validate user input with assertions, because it’s always difficult to make a parsable error message for the user.

For types in statically typed languages

If you use a statically typed language, you don’t need to use assertions for types, because the compiler will catch type errors for you.

For fulfilling non-critical failing

When you have an opportunity to introduce an assertion into a statement, but also want execution to persist even if the assertion fails, you should use a conditional statement instead of an assertion, then you can manage the situation is other way rather than using an assertion.

When does this turn into a solution for us?

When a programmer works on a program alone, he can easily track the state of the program and prevent it from entering into an invalid state. But when multiple programmers work on the same program, it becomes difficult to track the state of the program as the program grows, the functions are called from different places with different arguments, different parts of the program change and the behaviour not always is as expected. This is where assertions come into play. You’re lucky if you have time to write tests for your code, to check edge cases, but even thought it takes a lot of time to cover every edge case.

Assertions slow down the program?

Assertion underhood is just a simple if statement, so it doesn’t slow down the program, even though if you think it’s a problem for you and you expect to be the fastest program in the world, you can disable assertions in production, so many errors will be caught in testing environment.

Assertions replace tests?

Partially yes, but not completely. Assertions are used to check the state of the program, but tests are used to check the behaviour of the program.

At the same time the principle is similar to typings. If you say your function accepts number you don’t have to write tests to check if the argument is a number, that’s the responsibility of the caller to pass a number, not a string, in that case you are not testing the behaviour. The same with assertions, if you say your function accepts a number and you write assert to check if the argument is not zero, you don’t have to write tests to check if the argument is not zero, because you already know the result of that execution, it will throw an error, so you don’t have to test it.

How to handle errors produced by assertions?

You can handle errors produced by assertions the same way you handle errors produced by your code. You can catch them and handle them in a specific way, or you can let them bubble up to the caller.

For a backend application you can just log the error and return a 500 status code, for a frontend application you can show a message to the user and let him know that something went wrong (same applies to a mobile application or any other ui application).

Rules of thumb

Here is the list of rules for using assertions:

Use assertions to check for things that are very unlikely to happen, but if they do happen, they indicate a bug in your program.
Use assertions to document assumptions made in the code, making it more readable and maintainable.
Use assertions to check the state of a program and ensure it is behaving as expected.
Don’t use assertions for predictable error conditions (like invalid user input or file not found).
Don’t use assertions to handle or rectify errors. Their purpose is to highlight errors for easier debugging.

Personal opinion

Assertions is easy to use and very powerful tool, it helps to write more readable and maintainable code, it helps to prevent unexpected behaviour of the program. All the recommendations are very reasonable and I agree with them, but if you feel like you want to add an assertion which breaks the rules - just do it, the more you use them the better you understand their usage, it’s always better to add assertion to the place where it should be rather than not to add it at all.

Conclusion

In conclusion, assertive programming enables us to write robust, secure and efficient code by actively checking and validating state of a program. It’s about setting conditions that must be true at a certain point in our program and throwing an error if they are not. When used properly and responsibly, assertions can enhance the clarity and comprehensibility of our code, reduce debugging and maintenance time, and increase the overall quality of our software.

However, as with most tools, assertive programming has its limits. It’s not a replacement for well-thought-out control flow, validation of user input, or perceptive exception handling in general. Nor is it a replacement for a comprehensive test suite, as assertions and tests have their distinct roles.

Rather than seeing assertions as the panacea for our code quality, we should view them as an additional layer of defense, a form of “active documentation” that declares and enforces our assumptions about how functions are intended to behave. They complement other good coding practices, providing a more assertive and proactive approach to crafting reliable and maintainable software, and should be an important part of any developer’s toolkit.

Just like in the kitchen, “assertive” attention to detail can make all the difference between a successful meal-or a successful application-and a disastrous one. Happy assertive coding!

You don't need GraphQL to query less data

Fri, 24 Nov 2023 00:00:00 +0000

You don’t need GraphQL to query less data

Introduction

Imagine you’ve been working on the application for a long time. You’ve created many REST endpoints which are working fine, but you begin to notice that your queries are slowing down. You suspect it’s because you’re retrieving too much data from the server. You’ve heard about GraphQL and its way to query less data, but changing whole application structure to GraphQL is not best idea at that time, so you decide to remove redundant fields from the response, or create different endpoints for different client application view, yeah, it works, but it takes additional time to do that, it increases the number of bugs as you can forget to add some fields to some endpoints and so on. But is it really the only way? Let’s find out.

The problem

Let’s imagine that we have a simple REST endpoint which returns a list of users:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


[
 {
 "id": 1,
 "name": "John",
 "age": 20,
 "email": "john@mail.com",
 ...
 },
 {
 "id": 2,
 "name": "Jane",
 "age": 25,
 "email": "jane@mail.com",
 ...
 },
 {
 "id": 3,
 "name": "Bob",
 "age": 30,
 "email": "bob@mail.com",
 ...
 }
]

And we have a client application which uses this endpoint to display a list of users. The problem is that we don’t need all the fields from the response, we need only id, name, and email. So we have two options:

Remove redundant fields from the response.
Create a new endpoint which will return only the fields we need.

First way is the easiest one, but there is one big disadvantage:

At some point of the application you may need to grab age field was well as other fields.

Second way is a little harder, but it’s more flexible, but it also has a disadvantage:

You need to create a new endpoint for every view of the client application.
Once you add a new field to the user you need to add that to all endpoints where this field should be used.
Usually the team who works on the client application is different from the team who works on the server, so you need to communicate with them to add new fields to the endpoints.

Is there another way to resolve this issue without these disadvantages? Certainly, let’s look into it.

The solution

The solution is to use a query parameter which allows us to specify which fields we want to get from the server. Let’s call it fields, so now the request will look like:

For one view:

1

GET /users?fields=id,name,email

For another view:

1

GET /users?fields=id,name,age

Let’s define some rules and convention for that.

The rules

Here is the list of rules which, I believe, should be used for this approach:

If the fields parameter is not specified, return all fields.
If the fields parameter is specified, return only the fields which are specified in the parameter.
If the fields parameter is specified and the field is not found, just ignore that (do not throw an error or return value for that).
If the fields parameter consists only of fields which are not found, return an empty object.
If the fields parameter includes nested fields which are not found, but the parent field is found, return the parent field with an empty object as a value.
If the fields parameter includes nested fields but the actual value is not an object or array, do not return anything for that field (see example #5).

The convention

The last thing we need to think about is the convention for the fields parameter. I think that the best way is to have the fields parameter a string with comma-separated fields.

Before we start, let’s define the object we will have examples with:

1

GET /users?fields=id,name,email

Nested fields should be represented in the same way, but inside the parentheses, for example:

1

GET /users?fields=id,name,email,country(name)

The same logic applied to the arrays, for example:

1

GET /users?fields=id,name,email,phones(number)

Real examples

Let’s define the object we will have examples with:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


[
 {
 "id": 1,
 "firstName": "John",
 "lastName": "Doe",
 "username": "jdoe",
 "email": "john.doe@mail.com",
 "address": {
 "street": "123 Main St.",
 "city": "New York",
 "state": "NY",
 "zip": 10001
 },
 "dateOfBirth": "1980-01-02T00:00:00.000Z",
 "registered": true,
 "country": {
 "name": "United States",
 "code": "US",
 "information": {
 "capital": "Washington, D.C.",
 "population": 320000000
 }
 },
 "emergencyContacts": [
 {
 "firstName": "Jane",
 "lastName": "Doe",
 "phone": "212-555-1234"
 },
 {
 "firstName": "John",
 "lastName": "Smith",
 "phone": "212-555-1234"
 },
 {
 "firstName": "James",
 "lastName": "Johnson",
 "phone": "212-555-1234"
 }
 ],
 "roles": [
 "user",
 "admin"
 ]
 }
]

Example 1

Pick root fields:

1

GET /users?fields=id,firstName,lastName

Response:

1
2
3
4
5
6
7


[
 {
 "id": 1,
 "firstName": "John",
 "lastName": "Doe"
 }
]

Example 2

Pick nested fields:

1

GET /users?fields=id,country(name)

Response:

1
2
3
4
5
6
7
8


[
 {
 "id": 1,
 "country": {
 "name": "United States"
 }
 }
]

Example 3

Pick primitive fields from the array:

1

GET /users?fields=id,roles

Response:

1
2
3
4
5
6
7
8
9


[
 {
 "id": 1,
 "roles": [
 "user",
 "admin"
 ]
 }
]

Example 4

Pick nested fields from the array:

1

GET /users?fields=id,emergencyContacts(firstName,lastName)

Response:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


[
 {
 "id": 1,
 "emergencyContacts": [
 {
 "firstName": "Jane",
 "lastName": "Doe"
 },
 {
 "firstName": "John",
 "lastName": "Smith"
 },
 {
 "firstName": "James",
 "lastName": "Johnson"
 }
 ]
 }
]

Example 5

Pick nested fields from the primitive field:

1

GET /users?fields=id(value), roles

Response:

1
2
3
4
5
6
7
8
9


[
 {
 // id is not present in the response
 "roles": [
 "user",
 "admin"
 ]
 }
]

Extending the pattern

The rules for the pattern described above covers 99.9% of all cases, but there can be cases where you need to extend the pattern. Here is the list of possible extensions:

Regular expressions. For example, you want to get all fields which start with name or something like that. It’s not the best practice as it can be time-consuming as regular expressions are not the fastest thing in the world, but it can be useful in some cases.
Getting only the first (or last) N elements from the array. For example, you want to get only the first 5 elements from the array. It can be useful when you have a huge array and you want to get only the first elements to reduce the response size.
Get the defined list of fields of all objects inside some field (kind of instruments(*(id, name))).
Filtering by some condition (like order(*[ > 5])), but it’s rare case as it’s better to add that filter option right to the API.

Useful Libraries

Node.js

The library json-mask is a good choice for Node.js applications, it fully covers all the rules and conventions described above (except extended cases).

The library express-partial-response uses package json-mask under the hood and has prepared middleware for Express.js.

Python

The library jsonmask is a good choice for Python applications, it fully covers all the rules and conventions described above (except extended cases).

The library django-rest-framework-queryfields has similar functionality, but it is designed for Django REST Framework.

Search

Mon, 01 Jan 0001 00:00:00 +0000