Regular Expressions: Icebreakers

January 24, 2010

Few days back, there was a knowledge session on regular expressions within my team. After discussing the usual topics like greedy & lazy quantifiers, backreferences, etc, we started analyzing match results for few expressions. I ‘ve familiarity with regex and used them in majority of my throw-away scripts and I thought I knew regex unless been baffled with the simple questions from the team. I list down few of those simplest of the simple patterns and what they match and why ( which actually led to me learn the rules of the game),

Before even starting to look at them, did I mentioned earlier that regex engine would start its search just before the first character of the string ? If not, let me tell you now, it need to start before the first character, if and all the patterns  contains anchors ( ^, \b, etc ), it needs to check them too. And the search would go beyond the last character in the string and now you know why ( to match $, \b, etc ).

(i) x*

pattern : x*

string :foxxx

Matched: <<<  >>> foxxx

Explanation: As mentioned in the rule 2 here,  the greediness would always try to match more, hence read the pattern ‘x*’ as  ‘match more occurrence of x or nothing’. And the engine going to do its search character by character in the string. Since it could not find any ‘x’ to match at the starting position, it tries with its other choice ‘ match nothing’ and it succeeds.

(ii) .*

pattern : .*

string :foxxx

Matched: <<< foxxx >>>

Explanation: ‘.’ matches anything other than ‘\n’. Though the pattern ‘.*’ can be read as ‘match more of any characters other than ‘\n’ or nothing’, the rule of greediness gives the preference to match more characters.

(iii) x*

pattern : x*

string : xxxfoxxx

Matched: <<< xxx >>> foxxx

Explanation: Same greediness favors the match more criteria.

Advertisements

Regular Expressions: The Rules

January 24, 2010

The following are the rules, a non-POSIX regular expression engine(such as in PERL, JAVA, etc ) would adhere to while attempting to match with the string,

Notation: the examples would list the given regex(pattern) , the string tested against (string) and the actual match happened in the string in between ‘<<<‘ and ‘>>>’.

1. The match that begins earliest/leftmost wins.

The intention is to match the cat at the end but the ‘cat’ in the catalogue won the match as it appears leftmost in the string.

pattern : cat

string :This catalogue has the names of different species of cat.

Matched: This <<< cat >>> alogue has the names of different species of cat.

1a.The leftmost match in the string wins, irrespective of the order a pattern appears in alternation

Though last in the alternation, ‘catalogue’ got the match as it appeared leftmost among the patterns in the alternation.

pattern :species|names|catalogue

string :This catalogue has the names of different species of cat.

Matched: This <<< catalogue >>>  has the names of different species of cat.

1b. If there are more than one plausible match occurs in the same position, then the order of the plausible matching patterns in the alternation counts.

All three patterns have a possible match at the same position, but ‘over’ is successful as it appeared first in the alternation.

pattern : over|o|overnight

string :Actually, I’m an overnight success. But it took twenty years.

Matched: Actually, I’m an <<< over >>> night success. But it took twenty years.


2. The standard quantifiers (* +, ? and {m,n}) are greedy

Greediness (*,+,?) would always try to match more before it tries to match minimum characters needed for the match to be successful ( ‘0’ for *,? ; ‘1’ for + )

The intention is to match the “Joy is prayer”, though .* went pass across all the double quotes and grabbing all the strings only to match the last double quote (“).

pattern :”.*”

string :”Joy is prayer”.”Joy is strength”.”Joy is Love”.

Matched: <<< “Joy is prayer”.”Joy is strength”.”Joy is Love” >>> .

2a. Lazy quantifiers would  favor the minimum match

Laziness (*?,+?,??) would always try to settle with minimum characters needed for the match to be successful before it tries to match the maximum.

The first double quote (‘) appeared was matched using lazy quantifier.

pattern :”.*?”

string :”Joy is prayer”.”Joy is strength”.”Joy i
s Love”.

Matched: <<< “Joy is prayer” >>> .”Joy is strength”.”Joy is Love”.

2b. The only time the greedy quantifiers would give up what they’ve matched earlier and settle for less is ‘when matching too much ends up causing some later part of the regex to fail’.

The \w* would match the whole word ‘regular_expressions’ initially. Later, since ‘s’ didn’t have a character to match and tend to fail would trigger the \w* to backtrack and match one character less. Thus the final ‘s’ matches the ‘s’ just released by \w* and whole match succeeds.

Note: Though the pattern would work the same way without paranthesis, I’d used them to show the individual matches in $1, $2, etc.

pattern : (\w*)(s)

string :regular_expressions

Matched: <<< regular_expressions >>>

$1 = regular_expression

$2 = s

Similarly, the initial match ‘x’ by ‘x*’ was given by later for the favor of the last ‘x’ in the pattern.

pattern : (x*)(x)

string : ox

Matched: o<<< x >>>

$1 =

$2 = x


2c. When more than one greedy quantifiers appears in a pattern, the first greedy would get the preference.

Though the .* initially matched the whole string, the [0-9]+ would able to grab just one digit ‘5’ from the .*, and the 0-9]+ settles with it since that satisfies its minimum match criteriat. Note that the ‘+’ is also a greedy quantifier and here it cant grab beyond its minimum requirement, since already there is an another greedy quantifier shares the same match.

Enter pattern : (.*)([0-9]+)

Enter string : Bangalore-560025

Matched: <<< Bangalore-560025 >>>

$1 = Bangalore-56002

$2 = 5


3. Overall match takes precedence.

Ability to report a successful match takes precedence. As its shown in previous example, if its necessary for a successful match the quantifiers ( greedy or lazy ) would work in harmony with the rest of the pattern.


Sun CEO gets his wish

July 31, 2008

SEC OKs websites and blogs for Reg. FD, more here.


SunLDAP SSL configuration

August 12, 2007

  1. Create a CA root certificate:

Note: Either you can use the your existing CA root else follow these procedure for creating a CA root certificate

 2. Create a CSR for the SunLDAP server:

Using dsadm to create CSR for the SunLDAP server.  Login to the server installed with SunLDAP

# cd /opt/sun/ds6/bin            ### for RHAS 4 and /opt/SUNWdsee/ds6/bin for Solaris

#./dsadm request-cert –name ldaps://sun-ldapserver.com:636 –org SMI –org-unit SMI –city Bangalore –state Karnataka –country IN -F ascii -o /tmp/newreq.pem /var/opt/sun/dsins1   ### refer dsadm man for more info

# ls /tmp/newreq.pem
newreq.pem

Note: “-F ascii” is needed for the CA to sign the request

 3: Use the CA to sign the CSR:

Use the CA created in step 1 to sign the CSR

Note: Refer signing CSR using CA

 4: Enabling SSL for SunLDAP:

After completing the above procedures, we’ll be having the following certificates which needs to be added to the directory instance repository,

cacert.pem        –    CA Root certificate
newcert.pem        –    ldapserver certificate ( ldaps://hostname:port-no.)

1.Copy the certificates to the /tmp dir
2.Login to the server and change directory

    # cd /opt/sun/ds6/bin         ### for RHAS 4 and /opt/SUNWdsee/ds6/bin for Solaris

3.Add the server certificate
# ./dsadm add-cert <instance path>  <cert-alias> < certpath >
# ./dsadm add-cert /var/opt/sun/dsins1 server-cert /tmp/newcert.pem

4.Add the CA root certificate
# ./dsadm add-cert -C <instance path>  <cert-alias> < certpath >
# ./dsadm add-cert -C /var/opt/sun/dsins1 theCARoot  /tmp/cacert.pem

5.Set the server properties to use the newly added server certificate
# ./dsconf set-server-prop -e -p non-ssl-port ssl-rsa-cert-name:<certificate alias>
# ./dsconf set-server-prop -e -p 389 ssl-rsa-cert-name:server-cert          ### alias for server cert

6.Restart the DS for changes to get into effect
# ./dsadm restart instance path
# ./dsadm restart /var/opt/sun/dsins1

7.Now stop the DS and remove the default Cert (this ensures that the above generated certificate will be the default cert)
# ./dsadm stop instancepath
# ./dsadm stop /var/opt/sun/dsins1

8.Now remove the default certificate
# ./dsadm remove-cert instance-path cert-alias
# ./dsadm remove-cert /var/opt/sun/dsins1 defaultCert

 In case you want to export the above default cert, following is the command
# ./dsadm export-cert -o /<any path>/slapd-cert.export instance-path <original default cert alias>
where o=output file
9.Now start the server to use the ssl certificate created by us.
    # ./dsadm start instance-path
    # ./dsadm start /var/opt/sun/dsins1

10. Check using any ldap browser to the ports 389 and 636.


OpenLDAP SSL configuration – Part 2

August 9, 2007

If you are here through  CA root and ldapserver certificates creation mentioned earlier, please find the following name conventions used here,

cacert.pem      –   CA root certificate
newcert.pem   –   ldapserver certificate
newreq.pem   –   ldapserver key

Enabling SSL for Openldap:

The certificates can be moved to the desired certificate repository and renamed. Here I’ll be moving it to /usr/local/var/openldap-data

#cd /usr/local/var/openldap-data
#cp /opt/myca/demoCA/cacert.pem .
#mv /opt/myca/newcert.pem servercrt.pem
#mv /opt/myca/newreq.pem serverkey.pem

Add the following lines to slapd.conf:

*** Caution: Take a backup of the existing slapd.conf ***

#cd /usr/local/etc/openldap
#vi slapd.conf

TLSCipherSuite HIGH:MEDIUM:+SSLv2
TLSCACertificateFile /usr/local/var/openldap-data/cacert.pem
TLSCertificateFile /usr/local/var/openldap-data/servercrt.pem
TLSCertificateKeyFile /usr/local/var/openldap-data/serverkey.pem
# Client verification not required
TLSVerifyClient never

Start the openldap server by using the following command

# cd /usr/local/libexec
# ./slapd -h “ldap://ldapserver.com:389 ldaps://ldapserver.com:636”

Enter PEM pass phrase:< password >

Check to see the processes are listening in the configured ports

# netstat -an | egrep '389|636'
10.12.185.15.389 *.* 0 0 49152 0 LISTEN
10.12.185.15.636 *.* 0 0 49152 0 LISTEN

Use any external ldap-browser to connect to the both the URLs and check.


OpenLDAP SSL configuration – Part 1

August 9, 2007

To enable an SSL/TLS connection to the server, a server certificate is needed by the SSL/TLS protocols.  We are going to use openssl to create the certificates for our own testing purpose.

Procedure

1.Create a CA root certificate
2.Create a CSR for the ldap server
3.Use the CA to sign the CSR
4.Enabling SSL in openldap

Note: I’ve used Solaris10-x86 for the illustration

1: Create a CA root certificate<!–

Note:  When asked for a ‘Common Name’, you must enter the fully-qualified distinguished name of the server, e.g. myserver.com,

1. Create a directory for creating and signing your certificates.

#mkdir /opt/myca    

 2.  Change to /var/myca and run the OpenSSL CA script

# cd /opt/myca
# /usr/local/ssl/misc/CA.sh -newca
CA certificate filename (or enter to create)
Making CA certificate ...
Generating a 1024 bit RSA private key
...++++++
..........................................................................................................++++++
writing new private key to './demoCA/private/./cakey.pem'
Enter PEM pass phrase: < password >
Verifying - Enter PEM pass phrase:< password-again >   ### Remember the pass-phrase
-----
You are about to be asked to enter information that will be incorporated
into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:IN
State or Province Name (full name) [Some-State]:Karnataka
Locality Name (eg, city) []:Bangalore
Organization Name (eg, company) [Internet Widgits Pty Ltd]:SMI
Organizational Unit Name (eg, section) []:SMI
Common Name (eg, YOUR name) []:myserver.com
Email Address []:
Please enter the following 'extra' attributes to be sent with your certificate request
A challenge password []:
An optional company name []:
Using configuration from /usr/local/ssl/openssl.cnf
Enter pass phrase for ./demoCA/private/./cakey.pem: < password >
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 0 (0x0)
        Validity
            Not Before: Aug  6 17:08:44 2007 GMT
            Not After : Aug  5 17:08:44 2010 GMT
        Subject:
            countryName               = IN
            stateOrProvinceName       = Karnataka
            organizationName          = SMI
            organizationalUnitName    = SMI
            commonName                = myserver.com
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Comment:
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier:
                25:CB:8C:4A:5D:86:A6:41:84:01:08:47:5B:4B:63:E0:49:5B:58:BB
            X509v3 Authority Key Identifier:
                keyid:25:CB:8C:4A:5D:86:A6:41:84:01:08:47:5B:4B:63:E0:49:5B:58:BB

Certificate is to be certified until Aug  5 17:08:44 2010 GMT (1095 days)
Write out database with 1 new entries
Data Base Updated
#

This creates /opt/myca/demoCA/cacert.pem and /opt/myca/demoCA/private/cakey.pem (CA cert and private key). For our ldap configuration we ‘ll be using /opt/myca/demoCA/cacert.pem.

2. Create a CSR for the ldap server

Note:  When asked for a ‘Common Name’, you must enter the ldap url in the following format,
      ldaps://<FQDN of the ldap server>:<ssl-port>
     e.g. ldaps://ldapserver.com:636

# openssl req -newkey  rsa:1024  -nodes -keyout newreq.pem -out newreq.pem
Generating a 1024 bit RSA private key
.............................++++++
..++++++
writing new private key to 'newreq.pem'
-----
You are about to be asked to enter information that will be incorporated into your certificate request
What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank
For some fields there will be a default value, If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:IN
State or Province Name (full name) [Some-State]:Karnataka
Locality Name (eg, city) []:Bangalore
Organization Name (eg, company) [Internet Widgits Pty Ltd]:SMI
Organizational Unit Name (eg, section) []:SMI
Common Name (eg, YOUR name) []:ldaps://ldapserver.com:636
Email Address []:

Please enter the following 'extra' attributes to be sent with your certificate request
A challenge password []:
An optional company name []:

# ls
demoCA      newreq.pem

 

This will create a csr “newreq.pem”

3. Use the CA to sign the CSR

# /usr/local/ssl/misc/CA.sh -sign
Using configuration from /usr/local/ssl/openssl.cnf
Enter pass phrase for ./demoCA/private/cakey.pem:< password >
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 1 (0x1)
        Validity
            Not Before: Aug  6 17:10:02 2007 GMT
            Not After : Aug  5 17:10:02 2008 GMT
        Subject:
            countryName               = IN
            stateOrProvinceName       = Karnataka
            localityName              = Bangalore
            organizationName          = SMI
            organi zationalUnitName    = SMI
            commonName                = myserver.com
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Comment:
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier:
                7F:F3:DC:67:A9:48:60:D0:97:11:AE:F5:28:BF:8D:AC:4C:5B:DE:8C
            X509v3 Authority Key Identifier:
                keyid:25:CB:8C:4A:5D:86:A6:41:84:01:08:47:5B:4B:63:E0:49:5B:58:BB

Certificate is to be certified until Aug  5 17:10:02 2008 GMT (365 days)
Sign the certificate? [y/n]:y 

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=IN, ST=Karnataka, O=SMI, OU=SMI, CN=myserver.com
        Validity
            Not Before: Aug  6 17:10:02 2007 GMT
            Not After : Aug  5 17:10:02 2008 GMT
        Subject: C=IN, ST=Karnataka, L=Bangalore, O=SMI, OU=SMI, CN=ldaps://ldapserver.com:636
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (1024 bit)
                Modulus (1024 bit):
                    00:d6:d4:64:15:5a:7d:18:0e:23:cf:9b:40:8c:4a:
                    2a:5d:04:94:52:eb:a0:bb:ea:a1:40:3c:06:bd:e1:
                    1f:f8:8a:0f:ca:a6:0c:c8:22:5c:f4:e3:35:8a:6b:
                    48:81:6c:a1:8c:fc:b4:82:99:00:f6:a9:71:29:20:
                    68:e4:4c:84:87:48:34:e1:c8:78:59:41:57:09:ad:
                    2b:76:73:a3:0b:29:14:3d:0b:fc:96:7e:c6:51:99:
                    43:a8:9f:4f:13:95:cb:34:ba:fb:70:6c:d0:3a:ae:
                    65:0e:0a:5e:d3:cd:f9:20:9f:da:26:a4:35:bb:38:
                    dd:e2:46:93:6d:72:31:95:1d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Comment:
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier:
                7F:F3:DC:67:A9:48:60:D0:97:11:AE:F5:28:BF:8D:AC:4C:5B:DE:8C
            X509v3 Authority Key Identifier:
                keyid:25:CB:8C:4A:5D:86:A6:41:84:01:08:47:5B:4B:63:E0:49:5B:58:BB
 

    Signature Algorithm: sha1WithRSAEncryption
        6e:17:42:d8:f0:ee:f4:63:37:08:05:9e:6f:83:ed:d9:db:45:
        c0:60:2f:f0:06:51:bb:74:b9:bc:b1:1d:95:6e:0b:e9:98:39:
        93:ce:76:d1:16:a6:ea:c8:8b:50:ee:99:d6:5f:df:11:80:b1:
        3b:4c:7f:8c:3d:b3:3e:8b:a8:be:68:46:1c:6f:87:05:93:4d:
        d6:ca:1e:4d:c0:70:d4:b5:2d:fc:be:c4:8b:ba:20:35:94:32:
        e7:13:3e:7b:28:5e:98:28:02:d4:42:be:26:c2:08:d0:f0:3e:
        c1:20:fc:e7:1f:38:1d:69:d5:bf:84:e2:94:98:a8:05:ec:b4:
        20:66
-----BEGIN CERTIFICATE-----
MIICwTCCAiqgAwIBAgIBATANBgkqhkiG9w0BAQUFADBeMQswCQYDVQQGEwJJTjES
MBAGA1UECBMJS2FybmF0YWthMQwwCgYDVQQKEwNTTUkxDDAKBgNVBAsTA1NNSTEf
MB0GA1UEAxMWaWVjY2FwczIuaW5kaWEuc3VuLmNvbTAeFw0wNzA4MDYxNzEwMDJa
Fw0wODA4MDUxNzEwMDJaMHIxCzAJBgNVBAYTAklOMRIwEAYDVQQIEwlLYXJuYXRh
a2ExEjAQBgNVBAcTCUJhbmdhbG9yZTEMMAoGA1UEChMDU01JMQwwCgYDVQQLEwNT
TUkxHzAdBgNVBAMTFmllY2NhcHMyLmluZGlhLnN1bi5jb20wgZ8wDQYJKoZIhvcN
AQEBBQADgY0AMIGJAoGBANbUZBVafRgOI8+bQIxKKl0ElFLroLvqoUA8Br3hH/iK
D8qmDMgiXPTjNYprSIFsoYz8tIKZAPapcSkgaORMhIdINOHIeFlBVwmtK3Zzowsp
FD0L/JZ+xlGZQ6ifTxOVyzS6+3Bs0DquZQ4KXtPN+SCf2iakNbs43eJGk21yMZUd
AgMBAAGjezB5MAkGA1UdEwQCMAAwLAYJYIZIAYb4QgENBB8WHU9wZW5TU0wgR2Vu
ZXJhdGVkIENlcnRpZmljYXRlMB0GA1UdDgQWBBR/89xnqUhg0JcRrvUov42sTFve
jDAfBgNVHSMEGDAWgBQly4xKXYamQYQBCEdbS2PgSVtYuzANBgkqhkiG9w0BAQUF
AAOBgQBuF0LY8O70YzcIBZ5vg+3Z20XAYC/wBlG7dLm8sR2VbgvpmDmTznbRFqbq
yItQ7pnWX98RgLE7TH+MPbM+i6i+aEYcb4cFk03Wyh5NwHDUtS38vsSLuiA1lDLn
Ez57KF6YKALUQr4mwgjQ8D7BIPznHzgdadW/hOKUmKgF7LQgZg==
-----END CERTIFICATE-----
Signed certificate is in newcert.pem

# ls
demoCA       newcert.pem  newreq.pem

 

This creates newcert.pem (server certificate signed by CA) with private key, newreq.pem

So far we have created only the certificates needs to be used by the openldap server. Lets see how to configure the openldap using this certificates …

 


Configuring SunLDAP instances …

July 30, 2007

<!–Assuming you’ve already installed either ds6 or the whole JES5 suite, you would be having default server instance got installed under /var/opt/sun/dsins1 ( linux ). You might need to create more instances with non-default parameters eg. DN and the ports, as per your requirements.

Doing so in ds6 commandline requires you to tap your <enter> key only twice

# pwd
/opt/sun/ds6/bin
# ./dsadm create -p 1389 -P 2345  -D cn=Manager /var/opt/sun/dsins2
Choose the Directory Manager password:< Type a password for your new DN >
Confirm the Directory Manager password:< Re-type >
Use 'dsadm start /var/opt/sun/dsins2' to start the instance
# ./dsconf create-suffix -p 1389 -D cn=Manager dc=india,dc=sun,dc=com
Certificate "CN=<your host-name>, CN=2345, CN=Directory Server, O=Sun Microsystems" presented by the server is not trusted.
Type "Y" to accept, "y" to accept just once, "n" to refuse, "d" for more details: Y
Enter "cn=Manager" password:

 Connect using any external ldap browser and populate your instance by importing the suitable ldif.