7. Robustness and Efficiency Evaluation

Navigation

• Software Test Suites
  • Methodology
  • Test for routes
  • Test for models
• Stress Test
  • Methodology
  • Results
  • Identified Bottlenecks
  • Improvements made

Software Test Suites

Methodology

Tests are done using phpunit and the files can be found in the tests sub-directory in the project root directory.

Tests for routes

Routes tested: 10/15. Three routes, namely login, register, and logout came from Laravel out-of-the-box, which don't need testing. The 15 routes are the custom routes that we created.

The routes are tested with mainly three different scenarios: - Test access as a guest user - Test access as a resource owner - Test access as a non-resource owner

Tests for models

Models tested: 5/7

The models are tested in unit test style: each of the attributes and methods defined on the model are tested whether they behave as expected.

Stress Test

Methodology

Tools used

Apache HTTP server benchmarking tool -- ab, a tool for benchmarking Apache HTTP server is used in this procedure.

Laravel-Debugbar was also used to profile the memory and SQL statements made for each of the routes.

Types of tests

1. Sequential: 40 requests
2. Concurrent: 40 requests with 20 concurrent connections

The number of requests chosen is 40 because 40 can be considered a statistically significant number. Meanwhile, 20 concurrent connections were chosen just as a rule of thumb

Special cases

Some of the routes were not testable using ab for the following reasons:

1. File upload
2. The route creates a unique resource (which means that POSTing the same data again and again will result in an error)
3. The route deletes a resource (same as #2)

Routes that were not testable using ab were tested manually using Laravel-Debugbar 5 times sequentially. The reason it's done 5 times is because Laravel-Debugbar's profiling is done manually. Although it is not statistically significant, it conveys the rough time per request taken for the routes. For instance, for uploading record, the average time per request is 3.2s, this information is still useful rather than no information at all. It kinds of give an idea of how slow/fast the average response will be.

Measurements

In each of the routes, these measurements were made:

1. Time per request (TPR) for sequential test
2. Time per request (TPR) for concurrent test
3. Number of SQL statements and total time taken (for one instance of request on second action, not the average of many)
4. Front-end memory consumed
5. Number of failed requests in either sequential/concurrent test

The total time taken for the SQL statements were measured in such a way because it's observed that the first action will take much longer total time for some reasons. If we were to do a statistical analysis on this, the time taken for the subsequent actions will dominate, which will make the average lean more towards to the second action's time taken. However, because Laravel-DebugBar's profiling is done manually, it's rather tedious to do a statistical analysis on the time taken for the SQL statements to execute. Moreover, the SQL queries accounts for a very small percentage of the total response time. The max total time of SQL statements execution so far is 38ms.

Results

First of all, there were two measurements that remain rather constant across the tests:

1. The front-end memory consumed: 2MB (after minification of assets)
2. Number of failed requests: 0.

Routes	Sequential TPR	Concurrent TPR	Number of SQL statements (time taken (taken once))	Comments
GET /	364ms	258ms	0	-
POST /login	346ms	258ms	1 (5ms)	-
POST /logout	380ms	245ms	2 (7ms)	-
POST /register	520ms	-	2 (8ms)	Not testable using ab, so tested manually
GET /dashboard	439ms	343ms	3 (5ms)	-
GET /dashboard/record_issuers/ <:record_issuer_id>	394ms	278ms	4 (6ms)	The record issuer has 3 records in it
GET /stats/<:record_issuer_id>/ <:month>	386ms	271ms	5 (9ms)	The stats is for 3 records over a period of 6 months
POST /dashboard/record_issuers	452ms	-	3 (8ms)	Not testable using ab, so tested manually
DELETE /dashboard/record_issuers/ <:record_issuer_id>	430ms	-	-	1) Not testable using ab, so tested manually; 2) Number of SQL queries and response time depends on how many resources depends on the record_issuer
POST dashboard/record_issuers/ <:record_issuer_id>/upload_record	3.2s	-	8 (14ms)	1) Not testable using ab, so tested manually; 2) Depends heavily on the file size and especially the number of pages. In this case, a PDF file of 435kB and 2 pages is used; 3) A percentage of the response time were consumed in the file receive and storing
GET dashboard/records/ <:record_id>/template	444ms	293ms	6 (7ms)	-
POST dashboard/records/ <:record_id>/template	1.25s	-	17 (38ms)	Not testable using ab, so tested manually
POST dashboard/records/ <:record_id>/values	421ms	-	8 (8ms)	Not testable using ab, so tested manually
GET dashboard/records/ <:record_id>	402ms	362ms	-	Number of SQL statements not retrievable using Laravel-DebugBar
GET dashboard/records/ <:record_id>/download	416ms	298ms	-	Number of SQL statements not retrievable using Laravel-DebugBar
DELETE dashboard/records/ <:record_id>	507ms	-	0	Not testable using ab, so tested manually

Identified Bottlenecks

1. POST dashboard/record_issuers/<:record_issuer_id>/upload_record has a very slow response time. This is due to ImageMagick's processing time (converting PDF->JPEG). The response time depends heavily on the file size and number of pages in that file. The time taken to convert one PDF page to a JPEG image was measured and it averaged around 1s/page. This bottleneck seemed to be inevitable.

   “Speed was never a major feature of IM, which places more emphasis on the quality of the images it generates. That is not to say that it can't transform images in a reasonable amount of time. It's just not blindingly fast. Because of this, IM can be slow to perform certain processing operations, especially when attempting to compress images into image formats that have limited capabilities.”
   Taken from: http://www.imagemagick.org/Usage/

2. POST dashboard/records/<:record_id>/template has the second slowest response time. Again, this is due to ImageMagick's processing time in cropping image. On the other hand, the OCR didn't take a long time after all. It's measured that for image cropping, it averaged around 125ms/crop.

Improvements Made

It seemed that the processing time taken by ImageMagick is inevitable. As such, we have implemented a loading spinner (also as part of UI improvement) so that users are aware that the system is processing their files, justifying the time taken for the processing. Without the spinner, users could get the impression that the system hangs.