3G Drive test for experiencing customer point of view

3G Drive test for experiencing customer point of view
Version 1.a

Table of Contents
A.    3G Benchmarking Methodology    3
1.    General    3
2.    Coverage benchmarking (idle mode)    3
3.    Voice benchmarking (Qos mode)    3
4.    Video call  (Qos mode)    4
5.    PDD benchmarking (PDD mode) & Interconnection    4
6.    Video streaming , R99, HSDPA and HSUPA data mode benchmarking    6
7.    SMS benchmarking (sms mode)    6
B.    Benchmarking Items (Include in the benchmarking report)    7
1.    Coverage benchmarking (idle mode)    7
2.    Voice and video    7
3.    PDD benchmarking (PDD mode)    7
4.    Video streaming, R99, HSDPA and HSUPA    7
5.    SMS benchmarking (SMS mode)    7
C.    KPI Benchmarking Formula    7
1.    Coverage benchmarking (idle mode)    7
2.    Voice and Video call  benchmarking (Qos mode) ,    8
3.    PDD benchmarking (PDD mode)    9
4.    Video streaming, R99, HSDPA and HSUPA data mode    9
5.    SMS benchmarking (SMS mode)    17
Country destination call    19


A.    Benchmarking product checking
a.    Preliminary:
•    Team should invite regional to supervise before check cctivity begins
•    The check product done with K600 mobile station on a running on tems investigation must version 8.0 for 3G network. for HSDPA using Huawei modem E220.
•    Check product test should be done at 5 specific place in a city with stationary mobile station in good radio condition for 35 times per simcard for 4 simcards:
•    Indosat- 3G
•    Indosat-IM2
•    Indosat-Matrix 3G
•    Indosat-Mentari 3G
•    Indosat-IM3 3G
•    If there is problem on the service that being tested, the team must inform indosat regional and headquarter immediately, in order to resolving the problem.
•    The command sequence for MS in tools should be done in parallel, not allowed in serial sequence.
•    Check product test will be use sample simcard that represent physical home location registry (HLR) with minimum 2 simcards for each logical HLR
•    All 5 simcards will have to call a specific destination as a summarized in the following table:
√ = Test for 35 times
Location update test    Indosat 3G    Indosat IM2    Matrix 3G    Mentari 3G    IM3 3G
Phone Number    0815-12345    0856-123456    0857-123456    0816-12456    0816-12456
Location update to 3G    √    √    √    √    √
Location update to 2G    √    √    √    √    √

     Originating number    Originating number    Originating number    Originating number    Originating number
Destination number    Indosat 3G    Indosat IM2    Matrix 3G    Mentari 3G    IM3 3G
                       
Phone Number                       
Voice test call                       
Indosat-mentari    √    √    √    √    √
Indosat-Im3 (510 01)    √    √    √    √    √
Indosat-Im3 (510 21)    √    √    √    √    √
Indosat-matrix    √    √    √    √    √
Fren    √    √    √    √    √
Telkomsel    √    √    √    √    √
Excelcomindo     √    √    √    √    √
Flexy    √    √    √    √    √
Starone    √    √    √    √    √
3    √    √    √    √    √
PSTN local    √    √    √    √    √
* PSTN long distance    √    √    √    √    √
** Call using voip 01016    √    √    √    √    √
*** International call using ‘+’ sign    √    √    √    √    √
*** International call using 008    √    √    √    √    √
*** International call using 001    √    √    √    √    √
Video test call                       
Indosat-mentari    √    √    √    √    √
Indosat-Im3 (510 01)    √    √    √    √    √
Indosat-Im3 (510 21)    √    √    √    √    √
Indosat-matrix    √    √    √    √    √
Telkomsel    √    √    √    √    √
Excelcomindo    √    √    √    √    √
3    √    √    √    √    √
SMS test                       
Indosat-mentari    √    √    √    √    √
Indosat-Im3 (510 01)    √    √    √    √    √
Indosat-Im3 (510 21)    √    √    √    √    √
Indosat-matrix    √    √    √    √    √
Fren    √    √    √    √    √
Telkomsel    √    √    √    √    √
Excelcomindo     √    √    √    √    √
Flexy    √    √    √    √    √
Starone    √    √    √    √    √
3    √    √    √    √    √
Emergency call                       
112    √    √    √    √    √
110    √    √    √    √    √
Data access                       
GPRS    √    √    √    √    √
3G    √    √    √    √    √
Internet access via HP    √    √    √    √    √
Video streaming                       
Video portal 136    √    √    √    √    √
wap.3g.indosat.com    √    √    √    √    √
Internet access via HP    √    √    √    √    √
I-ring                       
Registration    √    √    √    √    √
Change song    √    √    √    √    √
3G Feature                       
Video Call    √    √    √    √    √
Video Mail    √    √    √    √    √
Portal 3G    √    √    √    √    √
Video Dating    √    √    √    √    √
High speed Internet Access    √    √    √    √    √
Video Streaming    √    √    √    √    √
3G Services                       
i-TV Mobile    √    √    √    √    √
i-Movie    √    √    √    √    √
i-Music    √    √    √    √    √
i-Business    √    √    √    √    √
i-Sport    √    √    √    √    √
i-Games    √    √    √    √    √
i-Religion    √    √    √    √    √
i-Infotainment    √    √    √    √    √
i-Lifestyle    √    √    √    √    √

* Destination city will be included (more than 1 city, can be ask to regional people)   
** Destination city and the country will be included (more than 1 city, can be ask to regional)   
*** Destination country is already included on this paper (more than 1 country) and should be tested all
Destination number can be finding in last page

B.    3G Interoperator Benchmarking Methodology

1.    General
a.    Team drive test should invite regional to supervise before benchmarking activity begins.
b.    Access permit to do drive test indoor will be issued by region
c.    Simcards used with the following configuration:
i.    MS1- Indosat
ii.    MS2 - Telkomsel
iii.    MS3 - Excelcomindo
d.    Drive test should be done using types of measurement listed as follows:
i.    Idle mode (mobile)
ii.    Voice Qos mode (mobile)
iii.    Video call Qos mode (mobile)
iv.    Video streaming Qos mode (mobile)
v.    PDD mode (stationary)
vi.    Data mode R99, HSDPA and  HSUPA (stationary)
i.    ftp mode Upload and Download files
ii.    http mode
iii.    ping mode
vii.    SMS mode (stationary)
e.    The drive test benchmarking should be done using K600i mobile station on a PC running on tems investigation must version 8.0 connected to a GPS for coordinate data collection, for HSDPA using Huawei modem E220
f.    Only multipurpose vehicle will be used for benchmarking drive test
g.    The drive test routes will cover all cities and intercity roads which will be determined by indosat
h.    All of measurement shall use “in car” mobile station antennas
i.    All mobile station (hand phone) must located in the in the middle seat of vehicle, mobile station do not allow to put in the dashboard
j.    The benchmarking drive test should be performed between Monday and Saturday

2.    Coverage benchmarking (idle mode)
a.    Mobile station1 shall use Indosat simcards
b.    Mobile station 2 shall use Telkomsel simcards
c.    Mobile station 3 shall use Excelcomindo simcards
d.    Idle benchmarking test performed in mobile condition following routes
e.    The idle mode for each mobile station should be done without locking to any specific band

3.    Voice benchmarking (Qos mode)
a.    Mobile station 1 shall use Indosat simcards and originate a call to test number 555
b.    Mobile station 2 shall use Telkomsel simcards and originate a call to test number 888
c.    Mobile station 3 shall 3 shall use Excelcomindo simcards and originate a call to test number 818
d.    The Qos measurement will have call sequence with 60 seconds connected and 10 seconds idle or wait time
e.    The benchmarking drive test should be performed during working hours from 08.00 to 20.00 on local time
f.    Qos benchmarking test performed in mobile condition following routes
g.    The Qos mode for each mobile station should be done without locking to any specific GSM band

4.    Video call  (Qos mode)
a.    Mobile station 1 shall use Indosat simcards and originate video call to B# mobile station
b.    Mobile station 2 shall use Telkomsel simcards and originate video call to B# mobile station
c.    Mobile station 3 shall use Excelcomindo simcards and originate video call to B# mobile station
d.    The Video call Qos measurement will have call sequence with 60 seconds connected and 10 seconds idle for wait time
e.    The benchmarking drive test should be performed during working hours from 08.00 to 20.00 on local time
f.    Video call Qos benchmarking test performed in mobile condition following routes but B# should be stationary
g.    The Qos mode for each mobile station should be done without locking to any band


5.    PDD benchmarking (PDD mode)
a.    Post dial delay (PDD) test should be done at 5 specific place with stationary mobile
Station in good radio condition for 35 times per simcard for 3 simcards:
i.    Indosat
ii.    Telkomsel
iii.    Excelcomindo

b.    All 3 simcards will have to call a specific destination as a summarized in the following table:

    Originating number
Destination number    Indosat 3G
    Telkomsel 3G
    Excelcomindo 3G

Indosat 2G    √    √    √
Telkomsel 2G    √    √    √
Excelcomindo 2G    √    √    √
Indosat 3G    √    √    √
Telkomsel 3G    √    √    √
Excelcomindo 3G    √    √    √
PSTN local    √    √    √
PSTN long distance (Jakarta)    √    √    √
International call (Singapore) using ‘+’ sign    √    √    √


6.    Video streaming , R99, HSDPA and HSUPA data mode benchmarking
a.    Test performed in stationary condition, done in 5 different places on a city and minimum sample taken is 35 sampling
b.    The data mode shall disconnect before one sampling test will be done, therefore for every one sampling always get new pdp context and IP assignment
c.    Mobile station 1 shall use Indosat simcards,  Mobile station 2 shall use Telkomsel simcards, Mobile station 3 shall use Excelcomindo simcards and do the following test:
a.    Video streaming
b.    R99, HSDPA and HSUPA required test for ftp upload 5 MB file size on http://www.drivehq.com/
c.    R99, HSDPA and HSUPA required test for ftp download 5 MB file size on http://www.drivehq.com/
d.    R99, HSDPA and HSUPA required test for http web load :
•    Domestic Server
i.    http load: http://www.indosat.com
ii.    http load: http://www.telkomsel.com
iii.    http load: http://www.xl.co.id
•    Overseas Server
i.    http load: http://www.yahoo.com
ii.    http load: http://www.msn.com/
iii.    http load: http://www.bbc.co.uk/
e.    For R99, HSDPA and HSUPA required test for ping web :
•    Domestic Server
iv.    http load: http://www.indosat.com
v.    http load: http://www.telkomsel.com
vi.    http load: http://www.xl.co.id
•    Overseas Server
i.    http load: http://www.yahoo.com
ii.    http load: http://www.msn.com/
iii.    http load: http://www.bbc.co.uk/

7.    SMS benchmarking (sms mode)
a.    Every mobile station should be connecting with dual com port data cable for this SMS test. First com port will connect as MS and another com port will connect as DC cable.
b.    SMS test should be done at 5 specific place with stationary mobile station in good radio condition for minimum 35 times per simcard for 3 simcards:
c.    The mobile stations will send and receive sms into its own simcards number with following test (on Net):
i.    Mobile station 1 Indosat simcard will send to its own
ii.    Mobile station 2 Telkomsel simcard will send to its own
iii.    Mobile station 3 Excelcomindo simcard will send to its own

The SMS command sequence (*.csq) will work on DC cable, and every send sms message should be distinguished by its contents E.g.:
i.    content of indosat : “send sms indosat1”, “send sms indosat2”, “send sms indosat3” etc
ii.    content of telkomsel : “send sms telkomsel1”, “send sms telkomsel 2”, “send sms telkomsel 3” etc
iii.    content of excelcomindo : “send sms excelcomindo 1”, “send sms excelcomindo2”, “send sms excelcomindo3” etc

A.    Benchmarking Items (will be Included in report)

1.    Coverage benchmarking (idle mode)
a.    SAN CPICH RSCP
b.    Ec/No 
c.    Number of unique SAN (Serving-Active set-Neighbors)
2.    Voice and video
a.    CSSR (Call setup success rates)
b.    CDR (Call drop rates)
c.    CSR (Call success rates)
d.    Signal Quality (EC/No)
e.    BLER
3.    PDD benchmarking (PDD mode)
PDD time is measured by taking the time between channel requests and alerting
4.    Video streaming, R99, HSDPA and HSUPA
a.    Network Availability             [Streaming, R99 HSDPA and HSUPA]
b.    Attach success rates         [Streaming, R99 HSDPA and HSUPA]
c.    Attach setup time             [Streaming, R99 HSDPA and HSUPA]
d.    PDP context activate success rates     [Streaming, R99 HSDPA and HSUPA]
e.    PDP context activate time         [Streaming, R99 HSDPA and HSUPA]
f.    PDP context cut off rates         [Streaming, R99 HSDPA and HSUPA]
g.    Streaming Service accessibility rates [Streaming]
h.    Streaming Service access time     [Streaming]
i.    Streaming quality            [Streaming]
j.    Streaming Reproduction cut off rates     [Streaming]
k.    Streaming Reproducing start delays        [Streaming]
l.    Service accessibility rates             [R99, HSDPA, HSUPA]
m.    Service access time             [R99, HSDPA, HSUPA]
n.    IP Service Access ratio             [R99, HSDPA, HSUPA]
o.    IP Service access time              [R99, HSDPA, HSUPA]
p.    Mean data rate                 [R99, HSDPA, HSUPA]
q.    Data transfer cut off rate             [R99, HSDPA, HSUPA]
r.    Ping RTT                     [R99, HSDPA, HSUPA]

5.    SMS benchmarking (SMS mode)
a.    Sms send success rates
b.    Sms delivery time

B.    KPI Benchmarking Formula
1.    Coverage benchmarking (idle mode)
Measurement is done in idle mode drive test. It was measured by extracting the SAN CPICH RSCP from the layer 3 message and then exporting it into Text / MapInfo to calculate the signal strength for each mobile station.

2.    Voice and Video call  benchmarking (Qos mode) , as follows:
a.    CSSR (Call setup success rates)
Measurement is done in Qos mode drive test. Call setup will start when first RRC connection request message send by the mobile station and a successful call setup has been established when an alerting message being received by mobile station.
This information can be observed the layer 3 message.
Channel request is defined as a call attempt and alerting is defined as call setup in tems event definition. The CSSR formula is defined as following:
CSSR = ∑ Call setup / ∑ Call attempt

b.    CDR (Call drop rates)
Measurement is done in Qos mode drive test. Measured from call setup (alerting) until disconnect cause 16, a normal call clearing, known as normal call behaviors. A drop call will be pegged when there is no disconnect cause 16 received by mobile station and the mobile station goes to idle mode.
This is defined as call drop in tems investigation tems event definition. The CDR formula is defined as following:
CDR = ∑ Call drop / ∑ Call setup

c.    CSR (Call success rates)
Measurement is done in Qos mode drive test. It is derived from CSSR and CDR and CSR formula is defined as following:
CSR = [CSSR * (1-CDR)]

d.    Signal Quality (EC/No)
Measurement is done in Qos mode drive test. It was measured by extracting the Ec/No from the layer 3 message and then exporting it into Text/MapInfo to calculate the EC/No for each mobile station.

e.    BLER

3.    PDD benchmarking (PDD mode)
PDD time is measured by taking the time between channel requests and alerting in layer 3 messages. The average time for PDD will be defined as below:
PDD time = Average [Δ Time (Alerting – channel request)]
4.    Video streaming, R99, HSDPA and HSUPA data mode
 Benchmarking will be formulated as follow:
a.    Network Availability [Streaming, R99 HSDPA and HSUPA]
To see probably there is no packet switched network available in the cell currently used by customer

Network Availability = ∑ Successful attempt to access PS network /
   ∑ attempt to Access PS network
             
 can be observed in L3 messages
b.    Attach success rates [Streaming, R99 HSDPA and HSUPA]
To see probably that a subscriber can attach to network

Attach success rates = ∑ Successful attach attempt / ∑ attach attempt

Can be observed in signaling layer 3 from phone sending attach request message until phone receiving attach accept or reject message

c.    Attach setup time [Streaming, R99 HSDPA and HSUPA]
To see duration of time period that has been taken to attach UMTS PS network.

Average (Attach setup time) = Average (∑Time(Attach accept – Attach
request))

Can be observed in signaling layer 3 from hand phone being send Attach request message till receiving Attach accept message.

d.    PDP context activate success rates [Streaming, R99 HSDPA and HSUPA]
To see probability that the pdp context easy to activate or not.
    PDP context activate success rates = ∑ Successful PDP activation attempts /
         ∑ PDP context activation attempts

can be observed in signaling layer 3 from hand phone being send PDP context activation request message and hand phone receiving PDP context activation message.

e.    PDP context activate time [Streaming, R99 HSDPA and HSUPA]
To see duration of time period that has been taken to activate PDP context.
Average (PDP context activation time) = Average (∑ΔTime (PDP context
activation accept – PDP context Activation request))

Can be observed in signaling layer 3 from hand phone being send PDP context activation request message and hand phone receiving PDP context activation message.


f.    PDP context cut off rates [Streaming, R99 HSDPA and HSUPA]
To see the probability that a PDP context has been deactivated, without being initiated intentionally by user.
PDP context cut off rates = ∑PDP context deactivation (not initiated by user) / ∑successful activated PDP cont.
   
can be observed in signaling layer 3 starting from hand phone being receive PDP context activation accept message until hand phone receiving the last data packet containing content.  And the PDP context deactivation not initiated intentionally by the user.

g.    Streaming Service accessibility rates [Streaming]
To see the probability that a first data packet of the stream can be received by the phone When requested by user.

Service accessibility = ∑ Successful streaming attempts  / ∑ streaming attempts

Can be observed signaling layer 3 starting from RTSP setup sent to the streaming server until reception of first RTP data packet

h.    Streaming Service access time [Streaming]
To see duration of time period that has been taken from requesting the stream at the portal until the reception of the first stream data packet by hand phone.

          Average (Service access time) = Average  (∑ΔTime(first streaming package
– streaming request))

Can be observed signaling layer 3 starting from RTSP setup sent to the streaming server Until reception of first RTP data packet

i.    Streaming quality[Streaming]
To see the quality of the stream reproduction as assessed by the VSQI algorithm.
Streaming quality = VSQI score for entire clip

j.    Streaming Reproduction cut off rates [Streaming]
To see the probability that a successful started stream reproduction is ended by a cause other than the intentional termination by the user:

Reproduction cut off rates = ∑lost streaming reproduction / ∑total streaming
     productions

Can be observed signaling by reception of first data packet  untill RTSP TEARDOWN Sent to the streaming server

k.    Streaming Reproducing start delays[Streaming]
To see time elapsing from reception of the first stream data packet by the phone until the phone starts reproducing the stream.
   
Average (Reproduction start delay) = Δ Time (start of reproduction – first
data packet)
Can be observed signaling by reception of first data packet and streaming player signaling that the reproduction of stream has started

l.     Service accessibility rates [R99, HSDPA, HSUPA]
To see the probability that subscriber can access the service successfully, this KPI gathered for test ftp upload, ftp download, http and ping  :

Service accessibility = ∑ Successful attempts / ∑ attempts

Can be observed signaling layer 3 starting from connect on command socket
Until reception of first data packet containing content

m.    Service access time [R99, HSDPA, HSUPA]
To see duration of time period that has been taken to access services successfully, from dial up connection is established until first data packet is received, this KPI gathered for test of ftp upload, ftp download, http and ping
   
Average (Service access time) = Average (∑Δ Time (first packet – connect to
socket))

Can be observed signaling layer 3 starting from connect on command socket 
Until reception of first data packet containing contents


n.    IP Service Access ratio [R99, HSDPA, HSUPA]
To see probability that after successfully activating a PDP context, a subscriber can access the service, so the data transfer can be started. this KPI gathered for test of ftp upload, ftp download, http and ping

IP service Access ratio  = ∑successful IP access / ∑ service access attempts

Can be observed signaling layer 3 from first [SYN] sent until reception of the first data packet containing content

o.    IP Service access time  [R99, HSDPA, HSUPA]
To see the time period needed to establish a TCP/IP connection to FTP server, from sending the initial query to a server until the first data packet is received. this KPI gathered for test of ftp upload, ftp download, http and ping
       
Average (IP service access time) = Δ Time (reception of the first data packet  –
first [SYN] send)

Can be observed signaling layer 3 from first [SYN] sent until reception of the first data Packet containing content

p.    Mean data rate [R99, HSDPA, HSUPA]
To see the data rates measured throughput the entire connect time (application throughput) this KPI gathered for test of ftp upload, ftp download, and http

        Mean data rate = (file size x 8 ) / Δ Time (last data packet  – first data packet)

Can be observed signaling layer 3 from first data packet containing content until reception of the last data packet containing content

q.    Data transfer cut off rate [R99, HSDPA, HSUPA]
To see the probability that a data can not be completed when it has been started successfully. this KPI gathered for test of ftp upload, ftp download, and http

        Data transfer Cut off rate  = ∑incomplete data transfer / ∑successful started data
transfer

Can be observed signaling layer 3 from first data packet containing content until
reception of the last data packet containing content
r.    Ping RTT [R99, HSDPA, HSUPA]
To see the respond time of server and routing starting from sending data and receive back
i.    Ping response time
Average [Time (Event ping response delay)]
ii.    Ping success rates
∑ Ping success / ∑ Ping request


5.    SMS benchmarking (SMS mode)
Measurement is done in sms mode in stationary test. And done on Net (a Hand phone will act as a sender and a receiver of sms).
a.    Sms send success rates
Sms send start when channel request messages followed by CP-Data UL message send by the mobile station and successful delivered when a CP-Data  followed by channel release message being received by mobile station.
This information can be observed in layer 3 message.
Sms send success rates = ∑ Channel release / ∑ Channel request

b.    Sms delivery time
SMS delivery time is measured by taking the time between channel request and channel release in layer 3 messages.
Channel request time pegged before CM service request uplink of mobile station and channel release time pegged after CP-Ack uplink of mobile station
This information can be observed in layer 3 message.

Sms delivery time = ∑ Δ Time (Channel release – Channel request) / ∑ Channel
Request


Country destination call
Number    Country    Phone number        Number    City    Phone number
1    AUSTRALIA    61 414 201 401        1    Bandung   
2    AUSTRIA    43 1 7983113        2    Jakarta   
3    BANGLADESH    880 2 98900 75        3    Bogor   
4    BELGIUM    32 2 2193532        4    Surabaya   
5    BRUNEI DS    673 2411188        5    Malang   
6    CANADA    16042610888        6    Etc   
7    CHINA    86 21 2535029        7    Etc . . .    
8    EGYPT    20 22 676600        8       
9    FRANCE    33 326056007        9       
10    GERMANY    49 69 90942148        10       
11    GREECE    30 210 330 1193        11       
12    HONGKONG    85224244386        12       
13    INDIA    91 11 2331 8383        13       
14    IRAN    98 213293500        14       
15    ITALY    39 0636892033         15       
16    JAPAN    81 3 33475515        16       
17    MACAO    8538913033        17       
18    MALAYSIA    60 12 2040 733        18       
19    NETHERLANDS    31 30 2386420        19       
20    NEW ZEALAND    64 9 8345342        20       
21    NORWAY    47 22775999        21       
22    OMAN    968 24633096        22       
23    PAKISTAN    92 51 2278799        23       
24    PHILIPINES    63 2 8172131        24       
25    PORTUGAL    351 917365562        25       
26    RUSIA    7 421 2213976        26       
27    SAUDI ARABIA    966 26472311        27       
28    SINGAPORE    65 65322883        28       
29    SOUTH KOREA    82 513280083        29       
30    SPAIN    34 91 396 2679        30       
31    SRILANKA    94 11 2438458        31       
32    SWEDEN    46 8 108 210        32       
33    SWITZERLAND    41 1 2946700        33       
34    TAIWAN    886 7 3446612        34       
35    THAILAND    66 2 2342896        35       
36    TURKEY    90 212 5209898        36       
37    U.A.E    971 4 2022573        37       
38    UNITED KINGDOM    44 1344 602228        38       
39    U.S.A    1 303 3917777        39       
40    VIETNAM    84 88391 807        40